← Presentation home · Part 5 of 13
Retrieve, Augment, Generate
Deep dive
Retrieve, Augment, Generate
TL;DR
RAG is three steps. Retrieve the most relevant passages from your documents. Augment the question by bundling those passages with it. Generate an answer from that enriched prompt. The model is taking an open-book test instead of a closed-book one. Every answer is citable.
The one-sentence version
RAG (Retrieval-Augmented Generation) means the AI does not answer from memory. It pulls the matching passage from your documents first, then writes the reply from that text.
Read that sentence twice. It is the whole tutorial in one line. Everything that follows just unpacks it.
The open-book test analogy
This is the best teaching tool in the whole deck. If you only remember one mental model for RAG, make it this one.
A closed-book exam tests what you memorized. That is a stock language model, or a fine-tuned one, answering from training. A student under pressure who half-remembers will guess, and a confident guess looks the same as a real answer to anyone in the room. That is the source of AI hallucination, the model is genuinely trying its best from what it remembers, and what it remembers is fuzzy.
An open-book test is different. You still need to be smart. You need to find the right page and understand what it says. You are not relying on memory.
That is RAG. The model's job shifts from "remember everything" to "read well." Modern language models are very good at reading. They are not as reliable at remembering. RAG plays to the strength.
The three steps, unpacked
Step 1, Retrieve
Your question is compared against every chunk of your documents. The comparison is not keyword matching, it is a math comparison of meaning (we go deep on how this works in slide 11's deep-dive). The system picks the top 5 passages closest in meaning to your question.
Why not keyword matching? Because language varies. Someone might ask "how do I file a sharps incident report" while the policy says "needle-stick injury reporting procedure." A keyword search misses entirely. A meaning-based search nails it.
Plain English: it finds the five most relevant pieces of your documents, even if they do not share any exact words with your question.
Step 2, Augment
The AI receives your original question *plus* those five passages, bundled together into one enriched prompt. "Augment" just means the question got padded with relevant context before the model ever saw it.
A simplified version of what the model actually receives:
Here are some passages from our policy documents:
[Passage 1, from Policy H.03 Section 4.2]
[Passage 2, from Policy A.11 Section 1.1]
[Passage 3, from Policy H.14 Section 3.5]
...
Now answer this question using only the passages above:
"How do I report a needle-stick injury?"The model does not know any of this glue was added by software. From its point of view, it just received a prompt that happens to have the answer material sitting right there in it. It reads, it answers from what it read.
Step 3, Generate
The AI reads the passages and writes an answer from them, not from its training or memory. And because the passages came from real documents, every answer is citable.
The model can name the exact source policy and section ("According to Policy H.03, Section 4.2.1..."), so a human can verify the answer in about ten seconds. The reader does not have to trust the AI. They can check.
That citation detail is not a nice-to-have. It is the difference between "an AI said so" and "an AI said so and here is the page." In a clinical, legal, or policy setting, that gap is everything.
The mental model to leave with
| Approach | What the answer sounds like |
|---|---|
| Stock model | "I think I remember..." |
| RAG | "According to Policy H.03, Section 4.2.1..." |
One of those you can take to a compliance officer. The other you cannot.
Why this beats fine-tuning, in one sentence
Fine-tuning hopes the model will remember enough facts to answer correctly. RAG just hands it the page. The first is gambling. The second is engineering.
Where this can still go wrong
RAG is not magic. It fails in predictable ways, and we cover each one in later deep-dives:
- The retrieval picks the wrong passages. (We get into chunk size, embedding choice, hybrid search in slides 06–11.)
- The retrieved passages exceed the model's context window. (The 2048-token gotcha, slide 08.)
- The model still hallucinates inside the passages it was given. (Less common, but possible. Good prompting and citation discipline mitigate.)
The whole rest of the tutorial is "RAG, but done well, with the gotchas surfaced." This slide is the concept. The next ones are the engineering.
FAQ
Where do the five passages come from? Why five? Five is a default, tunable. It is a balance: enough context to answer well, not so much that you bury the model or blow past its context window. Slide 08's deep-dive gets into what happens when that number is set carelessly.
What if the answer is not in the documents at all? Then a well-built RAG system says so. That is literally test number two in the live demo, ask about a feature that does not exist and watch it admit the gap. The honesty is a feature.
Does the model still use its training at all? Yes, for language, reasoning, and how to structure an answer. It just does not use training as the *source of facts*. Think of training as "how to read and write," and retrieval as "what is true here."
Is this just search with extra steps? Sort of, yes. RAG is "do a smart search, then have a smart reader write up the result." That is most of what it is. The trick is doing both halves well.
If you only remember one thing
RAG turns a closed-book test into an open-book test. The model stops guessing from memory and starts reading from your documents.