What is RAG (retrieval-augmented generation)?

Retrieval-augmented generation (RAG) is a straightforward idea: before asking the model a question, go and find the documents that answer it, and include them in the prompt.

That's it. The model isn't recalling anything — it's reading material you just handed it and answering from that.

Why bother

A model's knowledge is fixed at training time, general, and impossible to audit. Your company handbook is none of those things. RAG closes the gap:

Current — retrieval hits a live index, so today's document gets used today.
Private — your data stays in your database and is passed in at query time.
Attributable — because you know which documents you supplied, you can cite them.
Correctable — fix the source document and the answer changes. No retraining.

The last two are why RAG dominates in business settings. An answer you cannot trace to a document is not much use in a support tool.

How it works

Split your documents into chunks.
Convert each chunk to an embedding and store it.
When a question arrives, embed it and retrieve the closest chunks.
Paste those chunks into the prompt alongside the question.
The model answers from the supplied text.

RAG vs. the alternatives

The three ways to make a model know something it doesn't:

	RAG	Fine-tuning	Long context
Best for	Facts that change	Format, tone, behaviour	One-off, self-contained documents
Update cost	Edit the document	Retrain	Not applicable
Can cite sources	Yes	No	Yes
Cost per query	Low — a few chunks	Low	High — you pay for everything, every turn
Handles private data	Yes	Yes, but baked in	Yes

The common mistake is reaching for fine-tuning to teach a model facts. Fine-tuning teaches a model how to behave, not what is true. If the answer lives in a document, retrieve the document.

What RAG does not fix

RAG grounds answers; it does not guarantee them. It fails in specific, predictable ways:

Retrieval misses. If the right chunk isn't retrieved, the model answers from memory anyway — and may hallucinate confidently. Bad retrieval is worse than no retrieval, because the answer looks sourced.
Chunking destroys context. A paragraph severed from its heading may be retrieved into a context where it means something different.
Similarity isn't relevance. A chunk about the right topic can state the opposite of what's needed. Embeddings handle negation poorly.
Conflicting sources. Given an outdated and a current policy, the model has no reliable way to know which wins unless you tell it.

Most "RAG doesn't work" complaints are retrieval problems wearing a costume. If the system is answering badly, look at what it retrieved before you blame the model.

What we addedA direct comparison of RAG against fine-tuning and long-context prompting, with the failure modes RAG does not fix.

This article was researched and drafted with AI assistance from the sources listed below, then checked and edited by Fiqhro Dedhen before publication. How we work.