RAG (Retrieval-Augmented Generation)

1. What is it?

Retrieval-Augmented Generation grounds an LLM in your data at inference time: instead of relying on what the model memorized during training, relevant context is retrieved from an index you control and injected into the prompt before generation. The model composes answers from your knowledge; it does not invent them from its weights.

2. Why does it exist?

Three problems force RAG into existence:

1. Knowledge cutoff — models only know what they were trained on. Enterprise knowledge (configurations, standards, historical decisions) is not in any public training set.
2. Hallucination — a model asked about what it does not know will still answer, fluently and confidently. Grounding in retrieved documents gives answers something to be checked against.
3. Currency and cost — retraining or fine-tuning on changing knowledge is slow and expensive. Updating an index is an ETL job.

The mental model I use: the LLM is a reasoning engine, not a database. Retrieval is the database’s job; generation is the language’s job. Confusing the two is the root of most bad RAG systems.

3. How does it work?

The canonical pipeline has two phases:

• Ingestion — documents → chunking → embedding → vector index (+ metadata).
• Inference — user query → query transformation → retrieval (vector and/or keyword, often hybrid) → optional reranking → context construction → generation → citations.

Everything interesting happens in the seams between those stages, which is exactly where teams that treat RAG as “embed a PDF, call an API” get burned.

4. Important concepts

• Chunking — how documents are split. Structure-aware chunking (respecting section, table, and configuration boundaries) beats naive fixed-size splitting in every system I have worked on.
• Embeddings — a semantic compression of text. Same content, different embedding models → dramatically different retrieval quality. Changing the model means re-embedding everything, so choose deliberately.
• Vector search — similarity search over embeddings (e.g., pgvector, Milvus). Fast, but purely semantic: it misses exact identifiers unless you design for them.
• Hybrid search — combining vector similarity with keyword/BM25. The moment your corpus contains IDs, error codes, or part numbers, hybrid stops being optional.
• Reranking — a second-stage model that reorders retrieved candidates for relevance. Cheap relative to the quality it buys.
• Context construction — what finally goes into the prompt: deduplication, ordering, metadata, and token budgeting. The least glamorous step with the highest variance.
• Citations — mapping generated claims back to sources. In regulated domains this is a

5. Production considerations

• Latency — retrieval adds a hop before generation; rerankers add another. Budget the full path, not the model.
• Access control — retrieval is a data-access surface. Enforce permissions inside retrieval, never by prompting the model to behave.
• PII and confidentiality — your vector index is a new copy of sensitive data with its own lifecycle. Treat it like a database, because it is one.
• Caching — semantic caches and query-result caching cut cost and latency, but stale caches in fast-moving corpora produce confidently wrong answers.
• Observability — log the retrieved context with every answer. When quality drops, the first question is always “what did we retrieve?”
• Evaluation — retrieval metrics (recall, precision) and generation metrics (faithfulness, answer correctness) are different measurements. Track both, separately.
• Failure modes — wrong chunk retrieved (bad chunking), right document not retrieved (index/coverage gaps), right documents ignored by the model (context construction), hallucination despite good context (generation).

6. How I approach it

I treat retrieval quality as the product. Before touching prompts:

1. Build a small, honest evaluation set of real questions with known-good answers.
2. Measure retrieval in isolation — is the right chunk even in the top-k?
3. Fix chunking and metadata before touching the model.
4. Only then tune generation: structure, citations, and refusal behavior.

The most valuable prompt in any RAG system is the one that says “if the context does not contain the answer, say so.” Users forgive “I don’t know”; they do not forgive confident wrongness.

7. Architecture

A production RAG path, generalized from systems I have built:

User query
   → query understanding (expand, classify, route)
   → hybrid retrieval (vector + keyword, metadata filtered)
   → reranking
   → context construction (dedupe, order, budget tokens)
   → grounded generation (structured, cited)
   → validation & guardrails
   → answer + citations

The ingestion side (source systems → ETL → chunking → embeddings → index) usually matters more than the inference side, and it is where most of the operational effort lives.

8. Common mistakes

• Naive chunking that splits tables, configs, or clauses mid-meaning.
• top-k tuned for demo fluency instead of answer correctness.
• No hybrid search on corpora full of identifiers.
• Enforcing permissions via prompt instructions.
• Treating embedding model choice as an implementation detail.
• Shipping without an evaluation set and calling vibes a quality bar.

9. Interview perspective

Expect these, and answer like an engineer, not a tutorial:

• “Why RAG instead of fine-tuning?” — freshness, provenance, per-domain corpora, and auditability of sources. Fine-tuning teaches style and method, not mutable facts.
• “How do you evaluate a RAG system?” — separate retrieval evaluation (hit rate, MRR over labeled chunks) from generation evaluation (faithfulness, correctness), with a fixed golden set and regression gates.
• “How do you handle conflicting sources?” — precedence rules encoded structurally (site-specific over global), not left to model judgment.
• “Tell me about a production incident.” — have a real story: what you logged, what you changed, what you now monitor. This is where senior candidates separate from tutorial readers.

10. Related topics

• Agentic AI — retrieval as a tool inside agent loops.

• MCP — standardized surfaces for retrieval tools.

• LLM Evaluation — measuring whether any of this works.

  requirement, not a nicety.