Teaching AI Something It Was Never Trained On

Teaching AI Something It Was Never Trained On

TL;DR

By the end of this tutorial, you'll watch a local AI answer detailed questions about a programming language that did not exist until I wrote it last week. Nothing has been trained or fine-tuned. It is all retrieval. No cloud. No API keys. No data leaving the room.

What this is about

RAG supplies relevant documents at question time so the model can answer from evidence it did not receive during training. The answer still needs to be checked.

Most RAG demos use documents the model has probably half-seen during training already (Wikipedia, popular manuals, big open datasets). That makes it impossible to tell whether the model is reading the document or just remembering it. The honest part of the demo gets lost.

So we removed all doubt. We built a real, working symbolic programming language from scratch. No English keywords. Every control structure is a symbol. No LLM has ever heard of it. If the model answers correctly, it answered from the documents we gave it. Full stop.

The three live tests

The whole session is built around three demonstrations. Hold this checklist in your head, everything else explains how these three moments are possible.

1. Ask a real question. Watch the AI cite the documentation and produce correct code.
2. Ask about a feature that does not exist yet. Watch it admit the gap instead of inventing.
3. Ask something totally off-topic. Watch it refuse cleanly.

Each one tests a different property of a healthy RAG system. The first tests retrieval quality. The second tests hallucination resistance. The third tests topical containment. Pass all three and the system is doing real work, not theater.

What you do not need to know going in

No prior AI knowledge required. You do not need to know how transformers work, what an embedding is, what fine-tuning is, what vector search is. We define each term the first time it appears. If you can use ChatGPT you can follow this.

What you will know coming out

The talk compares our fine-tuning experiments with a retrieval-based approach, then explains the local components and how they fit together.

Why this matters

This architecture was used for an internal document-answering project at the University of Toronto Faculty of Dentistry. Its local design gave the team control over where prompts and documents were processed. Institutional approval and appropriate safeguards still matter.

FAQ

Is this just ChatGPT with extra steps? No. ChatGPT is a cloud service. Everything here runs on one machine on local hardware. The model, the search, the documents, all of it.

Did you train a model for this? No, and that is the entire point. Training was the wrong tool. The next deep-dive explains why.

Will I be able to build this myself after? Yes, if you can install Docker and run a couple of terminal commands. The "10-minute setup" deep-dive (slide 07) is literally a copy-paste install.

If you only remember one thing

You are about to watch a model answer correctly about a thing it has never seen. The trick is not in the model. The trick is in what you give it to read.