Design a RAG system

Clarifying questions

• Which document sources are authoritative and how often do they change?
• Do answers need exact quotations, summaries, or both?
• What is the acceptable behavior when retrieval confidence is low?

Functional requirements

• Ingest and chunk documents with provenance.
• Retrieve candidate passages and generate cited answers.
• Collect feedback and route failed answers into evaluation sets.

Nonfunctional requirements

• Keep answer latency under 5 seconds for ordinary queries.
• Do not answer from documents the user cannot access.
• Make low-confidence behavior explicit instead of inventing an answer.

Scale assumptions

• Five million documents, 200 million chunks.
• 1,000 peak queries per minute.
• Documents update continuously from multiple source systems.

API sketch

• POST /v1/answer { query, userId, corpusIds } -> { answer, citations, confidence }
• POST /internal/ingest/document { sourceId, version, acl, bodyRef }

Data model

• documents(id, source, version, acl_hash, updated_at).
• chunks(id, document_id, ordinal, text_hash, embedding_id, citation_span).
• answer_events(query_id, retrieved_chunk_ids, model_version, feedback).

Architecture components

• Ingestion service extracts text, chunks, embeds, and writes a vector index.
• Query path performs lexical retrieval, vector retrieval, reranking, and answer generation.
• Evaluation jobs replay labeled queries against retriever and generator versions.

Bottlenecks

• Embedding backfills can lag behind document updates.
• Vector search can return plausible but unauthorized chunks if ACL filters are bolted on late.

Failure modes

• Retriever confidence low: return a refusal with closest source suggestions.
• Index update lag: show document freshness metadata in citations.
• Model regression: rollback model version and keep retriever logs for replay.

Observability

• Retrieval recall on golden queries, citation coverage, refusal rate, answer latency.
• Chunk freshness lag and unauthorized-hit prevention counters.

Security / privacy

• Apply ACL filters before generation and recheck citations before response.
• Avoid logging raw private queries where retention has not been reviewed.

Cost considerations

• Generation dominates per-query cost; reranking and long context add second-order costs.
• Embedding cost follows changed chunks, not only changed documents.

Tradeoffs

• Smaller chunks improve pinpoint citations but can lose context.
• Hybrid retrieval is more complex than vector-only retrieval but protects exact identifiers.

ML-specific concerns

• training / serving skew: ingestion-time chunking must match query-time citation spans.
• Evaluation must separate retriever recall, citation support, and answer helpfulness.
• Prompt, model, retriever, reranker, and corpus versions need lineage on every answer.

Rubric