Our approach / solution

• A local model runs on hardware we own and writes the answers
• Production: NVIDIA Nemotron-3.5-Lightning on a DGX Spark
  • Latest NVIDIA model that actually fits this box. Shipped August 2026. That is what is loaded
  • NVIDIA GB10, 128 GB unified memory
  • MoE (mixture of experts): many specialist sub-networks, only a few fire per token
  • About 30 billion parameters on disk, about 3 billion active per token
  • Hybrid: Mamba-2 plus MoE plus attention. Built to run on one Spark
  • NVFP4 (NVIDIA 4-bit floating point): Spark-native 4-bit. More squeezed than 8-bit
  • Engine: vLLM
    • A GPU server that keeps the model loaded
    • Streams tokens as they are written
    • Speaks the same chat protocol the browser already uses
  • Card: Nemotron-3.5-Lightning on Hugging Face
• Homelab: Qwen 3.6 Heretic on a Mac Studio M2
  • Qwen is from Alibaba
  • Reasoning model, and also MoE (mixture of experts), same idea as Lightning
  • 6-bit compression (more squeezed than 8-bit, smaller on disk)
  • Base: Qwen 3.6 35B-A3B · uncensored cut: Heretic
• Open WebUI is the chat in the browser
• RAG (retrieval-augmented generation): cheating on a test, on purpose. The model does not memorize well. Search the library. Write from what you found
• A proxy is a program in the middle of two other programs
  • Same idea as a reverse proxy in nginx
  • The chat window talks to our proxy. The proxy talks to the model. Protocol level
  • That middle hop searches our docs, glues the pages together, and forwards the packed question to Lightning in production, or Qwen in the homelab
  • Try it: github.com/wildcard-wizard/rag-proxy

You
  |
  v
+------------------+
| Open WebUI       |  chat in the browser
+--------+---------+
         |
         v
+------------------+
| RAG Proxy        |  middle hop (nginx idea)
|  1. cheat sheet  |  which pin, this topic
|  2. search       |  other markdown in the folder
|  3. pack prompt  |  pin + hits + your question
+--------+---------+
         |
         v
+------------------+
| vLLM             |  local model, already loaded
|  Lightning / Qwen|
+------------------+

• The corpus is just a folder of markdown files. That is the library
• A pin is one of those files, kept short
  • You can open it in an editor like any other doc
  • Nothing is hidden inside the model
  • Search still runs on the rest of the folder
  • The pin is extra: we always paste that whole file in when the question is about that topic
• How the proxy knows which pin: we gave it a short list of rules
  • Not another library. A cheat sheet the proxy reads before it talks to the model
  • Example rule: if the question is about a needlestick, always paste needlestick-pin.md
  • That list is a settings file for the proxy. You can think of it as a table: topic → file
• We did not add reranking
  • Search is a first guess. It returns the closest pages, in a rough order
  • Reranking is a second pass: a smaller model re-scores those pages against this exact question, then keeps the top few
  • Extra machinery. Extra GPU. And it cannot save you if the right page never made the first list
  • A pin skips that luck. Needlestick question → always paste needlestick-pin.md
  • The must-see page is in the prompt even if search ranked it fifth
  • Search still runs. We parked the second ranker. Pins are doing the job for the pages that must not be missed

corpus/                         the library (just markdown)
  Policy-H03.md                 normal doc. Search may find this
  clinic-hours.md
  needlestick-pin.md            also markdown. This is the pin

Cheat sheet the proxy reads:
  needlestick question  →  always paste needlestick-pin.md
  hours question        →  always paste clinic-hours.md

You ask about a needlestick. The proxy looks at that cheat sheet, pastes needlestick-pin.md onto the prompt. Search may still add Policy H.03. Two markdown files. One extra rule that says "this topic, this file."

# needlestick-pin.md

Use Policy H.03 only.
Do not invent a policy number.
If H.03 is not in the pages we found, say you do not have it.

More links: Open WebUI · vLLM · MLX