← Presentation home · Part 3 of 13
Why not fine-tuning?
Deep dive
Why not fine-tuning?
TL;DR
Fine-tuning is the obvious idea. Feed your documents to a model, it learns your domain. It does not work like that. Fine-tuning teaches style and format, not facts and recall. The model comes back sounding perfect and inventing facts confidently. I spent five months learning this the hard way.
The obvious wrong turn
The first time someone hands you a stack of documents and says "make an AI that knows this," the temptation is overwhelming: fine-tune a model on the documents. It is the kind of idea that sounds right. It is the idea every team tries first. I tried it. It is wrong.
Fine-tuning (continuing to train an existing model on your own examples so it adjusts its internal weights) feels like it should teach the model new facts. The mental model is: feed it your documents, it becomes an expert on your domain.
That mental model is wrong, and the way it is wrong is dangerous.
What fine-tuning actually changes
| Thing | Plain English |
|---|---|
| What fine-tuning *changes* | The model's style, tone, and format. It learns to sound like your documents. |
| What fine-tuning *does not change* | The model's facts and recall. It does not reliably learn what your documents say. |
The fine-tuned model came back sounding perfect. It wrote in the exact institutional voice of a Faculty of Dentistry policy document. Confident. Professional. On-brand.
And the facts were invented. It made up email addresses. It fabricated form names the Faculty has never used. It cited policies that do not exist. On one run, it refused to answer a question it had been explicitly trained on.
The one-line summary: the model learned how we talk, not what we know.
Why this happens (the intuition)
A language model's training does not store facts the way a database stores rows. Training nudges billions of internal weights toward patterns. Style is a strong, repeated pattern, every sentence in your corpus reinforces the tone, so style sticks fast. A specific fact (one email address, one form number) might appear once or twice. That is not enough repetition to move the weights in a reliable way.
The model picks up the music and loses the lyrics.
The next deep-dive (slide 04) has the hard research numbers behind this. This one is the lived experience. Slide 04 is the proof.
The dangerous part
It is not that fine-tuning fails. It is that fine-tuning fails while sounding right.
A confidently wrong answer in a clinical setting is worse than an obviously broken one. An obviously broken answer gets caught. A polished, professional, completely fabricated answer often does not. The institutional voice is itself a credibility signal, and the model has *just learned that exact voice*. So the wrong facts come wrapped in the most trustworthy possible packaging.
If you are evaluating an AI for any setting where the cost of being confidently wrong is high, healthcare, legal, finance, anywhere a real person acts on the output, this failure mode is the one to be most afraid of.
What I tried before giving up
Six runs, escalating effort:
- Vanilla fine-tune on the documents. Got the voice. Lost the facts.
- Different base model. Same result, different voice.
- Heavier training (more epochs, lower learning rate). Sounded *more* convincing while being just as wrong. Worse, not better.
- Better data prep (chunked, labeled, structured). Same.
- Aggressive Q&A pair generation, synthesize thousands of question/answer pairs from the docs, fine-tune on those. Still hallucinated.
- The most expensive setup I could justify. No improvement.
Five months. Same destination every time.
Is fine-tuning ever useful?
Yes, for the things it actually does. Use fine-tuning when you want:
- A house voice. Always respond in your brand's tone. ("Be friendly but not sycophantic. Use second person. Avoid jargon.")
- A strict output format. Always reply as JSON with these fields. Always start with a one-line summary. Always end with a citation block.
- A genre or domain dialect. Match the cadence and vocabulary of legal contracts, or medical case notes, or a specific support-doc style.
For all of those, fine-tuning is the right tool. It is the wrong tool for injecting facts.
What about fine-tuning *and* RAG?
It exists. The combination is called RAFT (Retrieval-Augmented Fine-Tuning, a technique from Microsoft) and slide 04 covers it. Short version: it works, but only while the retrieval step is running alongside. Strip away the retrieval and RAFT underperforms a stock model. The retrieval was doing the work. So just do retrieval properly and skip the training cost.
FAQ
Did you fine-tune it properly? Maybe the settings were off. That is exactly what the next deep-dive answers. Six runs, escalating effort, including the most aggressive configurations. The research shows the ceiling is real, not a tuning mistake.
Is fine-tuning ever useful? Yes, for style and format. If you want a model that always responds in a house voice or a strict output structure, fine-tuning is the right tool. It is just the wrong tool for injecting facts.
What about fine-tuning plus RAG together? That combination exists (RAFT). It works, but it reintroduces the document-fetching step, so you might as well do RAG properly and skip the training cost.
Could a much bigger model fix this? No, the next deep-dive cites Stanford's FineTuneBench, which showed a 37% generalization ceiling regardless of method or model size. Scale does not solve a repetition problem.
If you only remember one thing
Fine-tuning fails while sounding right. A confident wrong answer in a clinical setting is worse than an obvious broken one, because nobody catches it.