What is RAG?

The four steps, and where the work is

Chunk: split your documents into passages small enough to sit in a prompt and large enough to make sense alone. Index: make those passages findable, by embedding them, by keyword index, or both. Retrieve: at request time, turn the question into a query and pull back the top passages. Generate: build a prompt containing the question, the passages and an instruction to answer only from them and to cite which one was used.

Read that list again and notice how little of it involves the model. Three of the four steps are search engineering, which is the single most useful thing to internalise about RAG: it is a retrieval system with a language model at the end, and teams that treat it as a model problem spend months tuning prompts against a bad index.

Where RAG actually breaks

Almost always in retrieval, and almost always in one of four ways. Chunking that splits a table, a code block or a definition from its heading, so no single passage carries the answer. Pure vector search with no keyword fallback, which fails exactly on error codes, flags and version numbers where users are most specific. A stale index, because nothing re-embedded the pages that changed. And too many passages, which pushes cost up and, in a long enough prompt, buries the relevant one among near duplicates.

Each of those has a boring fix: chunk along document structure rather than character count, run keyword and vector search together and merge, rebuild on content change rather than on a schedule, and rerank a small final set instead of pasting twenty candidates. All of them are measurable before a model is involved. Evaluate retrieval on its own, by checking whether the passage containing the answer is in the top results at all, and most of the mystery leaves the project.

RAG, fine-tuning, memory and long context

Four different questions. RAG changes what the model knows for this one request, and is the right tool for facts that change and must be cited. Fine-tuning changes how the model behaves, format, tone, a narrow classification task, and is a poor and expensive way to teach facts. Memory is state your system carries between sessions about a user or a project, which is often implemented with retrieval but answers a different question. Long context means pasting more in directly, which is simpler and works well until the corpus is bigger than the window or the bill.

They compose rather than compete. A common shape is a fine-tuned small model for classification, retrieval for the facts, and a long context window for the handful of documents that must be present in full.

Where the term came from

The name comes from a 2020 paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, which described a specific architecture: a trained retriever over a dense index combined with a sequence to sequence generator, tuned together.

Almost nobody means that now. In current usage RAG describes the pipeline shape, retrieve then prompt, with no training involved anywhere and off the shelf embedding or keyword search doing the retrieval. Worth knowing when you read the literature, because papers about RAG frequently discuss the trained variant while every product blog means the pipeline.

q = "what is the log retention on the Team plan?"

# 1. two searches, merged: vectors for phrasing, keywords for exact terms
hits = merge(vector_search(q, k=20), keyword_search(q, k=20))
top  = rerank(q, hits)[:5]          # small final set, not twenty

# 2. the prompt, with the contract stated explicitly
prompt = f'''Answer ONLY from the passages. Cite as [n].
If they do not contain the answer, reply: NOT IN SOURCES.

{format_passages(top)}

Question: {q}'''

# 3. evaluate the halves separately
#    retrieval: is the passage holding the answer in `top` at all?   (recall@5)
#    generation: is every claim traceable to a cited passage?        (faithfulness)