← Presentation home · Part 4 of 13
The research backed up the failures
Deep dive
The research backed up the failures
TL;DR
Three independent research findings confirm what I learned the hard way: fine-tuning is not the tool for fact injection. RAG hits 87.5% accuracy where fine-tuning hits 50.4%. Single facts need 100–1,000 repetitions to stick through training, but most documents mention any given fact maybe 15 times. The model's own makers tell you to use RAG.
What this is
After five months of fine-tuning failures, I went and read the research. Turns out I was not the first. The pattern was documented. The numbers were public. The model creators themselves were quietly recommending exactly what I had been refusing to consider.
Three findings, each one independent, each one pointing the same direction.
Finding 1, Microsoft EMNLP 2024: RAG 87.5% vs Fine-Tuning 50.4%
Microsoft ran a controlled comparison. Same questions, same domain, same evaluation. RAG scored 87.5% accuracy. Fine-tuning scored 50.4%.
The plain-English read: fine-tuning for fact recall is barely better than a coin flip. RAG nearly doubles it.
This is not a small margin you can tune away with better hyperparameters. It is a structural gap. Two completely different approaches, one of them works for knowledge tasks, one of them does not, and the size of the gap is the size of "different category of solution."
The paper is published at EMNLP (Empirical Methods in Natural Language Processing) 2024. It is not a preprint, not a blog post, not a single team's anecdote. It went through peer review at one of the top NLP conferences.
Finding 2, Allen-Zhu and Li, ICLR 2025: facts need 100 to 1,000 exposures
This study explains *why* the gap from finding 1 exists.
For a language model to reliably learn a single fact through training, it needs to see that fact between 100 and 1,000 times, in varied contexts. Not 100 to 1,000 times across the whole corpus, 100 to 1,000 times for *each individual fact*.
Now do the math on real documents. The Faculty's policy documents mention any given fact (an email address, a form number, a specific procedure) maybe 15 times across the whole corpus. That is off by two orders of magnitude from what training needs.
The training never had a chance. It is not a tuning problem. It is an arithmetic problem. You cannot fine-tune your way around a number that small.
The paper is published at ICLR 2025 (International Conference on Learning Representations). It is methodologically careful, the researchers built synthetic datasets where they could control exactly how many times each fact appeared, then measured when the model would reliably recall it. The 100–1,000 range is the empirical threshold.
Finding 3, Phi-4 Mini model card: the makers themselves say use RAG
The most quietly damning finding.
Phi-4 Mini is a small, capable model from Microsoft. When you download a model, it comes with a "model card", the official documentation describing what the model is good for, what it is not, and how to use it well.
Phi-4 Mini's model card recommends RAG over fine-tuning for knowledge tasks.
When the model's own creators, the people who built it, who trained it, who know what it can and cannot do, tell you not to fine-tune for facts, that is the ballgame. There is no clever configuration that the model creators missed. The advice is in the manual.
Two more findings worth knowing
The slide focuses on three. Two more strengthen the case:
- Stanford FineTuneBench showed a 37% generalization ceiling regardless of method or model size. Even fine-tuning done perfectly hits a wall. Scale does not save you.
- Meta's LIMA paper introduced the "Superficial Alignment Hypothesis", the formal claim that fine-tuning teaches format and tone, not new knowledge. That is exactly the lived experience from slide 03, stated as research.
Five independent sources. One conclusion.
The progression at the bottom of the slide
The arrow shows the journey: Fine-Tune → RAFT → RAG → RAG Proxy. That is the whole talk in four words.
| Stage | Plain English |
|---|---|
| Fine-Tune | The first attempt. Teaches style, not facts. Failed. |
| RAFT | Retrieval-Augmented Fine-Tuning. Works, but it needs a retrieval step alongside it. If retrieval is doing the work, why train at all. |
| RAG | Retrieval-Augmented Generation. Hand the model the documents at question time. The pivot that worked. |
| RAG Proxy | My build. RAG as a piece of infrastructure instead of a feature locked in one app. |
Every slide after this one is somewhere on that arrow.
Why these papers matter, even if you do not normally read papers
If someone in your organization pushes back on "we cannot fine-tune our way out of this," these are your receipts. Four independent research groups, Microsoft, two academics at ICLR, Stanford, Meta, all pointing at the same finding through different methodologies.
It is not your team that failed. It is not your data. It is the technique. The technique was the wrong tool for the job, and the literature said so before you started.
The frustrating part: the papers were available the whole time. The lesson I took out of this: check the literature before you spend five months on something. The problem you are working on has probably already been characterized.
FAQ
Could a bigger model fix this? Stanford's FineTuneBench says no. The 37% ceiling held regardless of model size. Scale does not solve a repetition problem.
What is RAFT exactly? Retrieval-Augmented Fine-Tuning, a Microsoft technique that trains the model to read documents alongside the question. It scores well, but only while a retrieval step runs alongside it. Strip the retrieval away and it underperforms a stock model. The retrieval was doing the work, so just do retrieval.
Are these papers cherry-picked? They come from independent groups using different methodologies, controlled benchmarks, synthetic corpora, model-creator documentation, generalization studies, and they point the same direction. Four groups, one conclusion. That is the opposite of cherry-picking.
If I have already started a fine-tuning project, should I stop? Depends on what you are fine-tuning for. If it is style, format, voice, output structure, keep going, that is what fine-tuning is for. If it is facts and recall, the literature says you are going to hit a wall. RAG is the pivot.
If you only remember one thing
Five independent research groups, one conclusion: fine-tuning teaches style, not facts. Stop trying to put knowledge in the model. Put it in a layer.