The research backed up the failures

What the research supports

Fine-tuning and retrieval solve different problems. Neither guarantees accurate answers. For a changing document library, retrieval makes it easier to supply current passages and show where an answer came from.

FineTuneBench: a measured limitation, not a universal ceiling

FineTuneBench (Wu, Wu and Zou, 2024) evaluated five commercial fine-tuning APIs. It reported average generalization accuracy of 37% for new knowledge and 19% for knowledge updates in its experiments. Those are benchmark averages, not a maximum that applies to every model, dataset or training method.

RAFT: training and retrieval can work together

RAFT (Zhang and colleagues, 2024) trains a model to answer using relevant retrieved documents while ignoring distracting ones. The paper reports improvements on its evaluated domain-specific tasks. It supports combining fine-tuning with retrieval when evaluation justifies the extra work.

What this means for the project

Our fine-tuning experiments produced convincing wording without dependable policy recall. We moved to retrieval so each answer could be checked against the supplied documents. That is a project decision, not proof that fine-tuning can never teach facts.

• Start with a small set of real questions and expected answers.
• Check retrieval quality separately from answer quality.
• Test missing-document and off-topic questions as well as successful ones.
• Compare approaches on the same questions before choosing a more complex system.

The original slide is retained as a record of the talk. These revised notes replace its unsupported percentage comparison and repetition threshold.