Lessons learned

• I went on a fine-tuning journey first
  • Fine-tuning means extra training on a model that already exists
  • Like adding bookmarks in a book you already own. You are not reprinting the book. You are marking "talk like this"
  • It taught style, not the answer. It sounded like our shop. It did not recall the procedure
  • RAG (retrieval-augmented generation) put the facts in the prompt, this question, from our files
• Pins again
  • The corpus is a folder of markdown files
  • A pin is one of those files. A cheat sheet tells the proxy: this topic, always paste this file
  • Search still runs on the rest
  • Reranking would re-score search hits with a second model. We parked it. Pins already put the must-see file in
• A harness is software that helps guide the LLM (large language model)
  • My version runs after the model talks
  • The prompt is a request. The harness is a check. The model cannot sweet-talk the checker
  • Think of a compiler: you do not trust the author because they sound confident. You run perl -c on the program. If Perl says the syntax is broken, it is not done
• How we check clinic answers
  • We keep a list of real policy numbers from our library
  • If the model cites Policy H.99 and H.99 is not on that list, we stop. Fluent is not the same as true
  • If the retrieved hours say Monday to Friday and the model adds Saturday, that Saturday did not come from our pages. The harness can catch that
  • Same idea for TOAD or Perl: if the answer includes a program, we run the real language check (toad -c or perl -c) on that program
• Weak model, strong cage
  • The LLM is a guesser, even when it is huge
  • We do not need it to be a lawyer. We need retrieve, pin, then software that the model cannot charm
  • Small or large, we own the checker

Proxy: github.com/wildcard-wizard/rag-proxy