Why use local AI?

Deep dive

Why use local AI?

TL;DR

Three reasons: privacy, no query tracking, and the legal one, PHI (Personal Health Information) cannot leave the building. A single PHI breach runs $500,000 to $1,000,000. Local AI keeps every prompt, every document, and every answer behind your own firewall.

The skeptic's question

Before any technical content, let's answer the question someone is already thinking: why not just use ChatGPT? It is better than anything you can run at home. For a casual question, fine. For drafting a policy, asking about a patient scenario, or working through internal infrastructure decisions, the calculus changes.

Local AI is not always the right answer. It is the right answer when the data matters.

Reason 1, Privacy: your data is your intellectual property

When you send a prompt to a cloud AI (ChatGPT, Claude, Gemini, take your pick), that text leaves your building and lands on someone else's server. The vendor's policies are usually reasonable. They are also subject to change at any time, and they cannot retroactively unsend what you already sent.

For a casual "explain this regex" question, none of that matters. For a draft of an unpublished policy, a patient scenario rephrased to remove names, internal infrastructure notes that describe how your network is segmented, you have just handed your intellectual property to a third party. There is no "undo."

Local AI keeps every prompt and every document behind your firewall. The model runs on hardware you own. Nothing is transmitted. The privacy property is not a promise from a vendor; it is a property of the architecture.

Reason 2, Data tracking: every cloud query is logged

This is not a conspiracy claim. It is in the terms of service. Cloud vendors log queries. Those logs can be analyzed, retained, and in many cases used to improve the vendor's future models. Your prompts can become training data for somebody else's product.

Some vendors offer "do not train on my data" toggles. Some offer enterprise tiers with stricter retention guarantees. All of those are policies on top of a system that, by design, does see your data. Removing the cloud removes the data-handling problem entirely.

With a local setup there are no external logs because there is no external anything. The only record is whatever you choose to keep on your own disk.

Reason 3, PHI: it legally cannot leave the building

This is the one that decides things for healthcare.

PHI (Personal Health Information, anything that identifies a patient and their health status) is governed in Ontario by:

• PHIPA (Personal Health Information Protection Act)
• FIPPA (Freedom of Information and Protection of Privacy Act)

Patient-adjacent data cannot be sent to a cloud service. Not "should not." Cannot. Cloud AI was never an option for clinical use at the Faculty of Dentistry. Other jurisdictions have their own versions, HIPAA in the US, GDPR in Europe, and the same principle holds: certain categories of data are legally fenced.

The cost of getting this wrong

The slide cites a PHI privacy breach at $500,000 to $1,000,000. Use this number out loud the next time someone proposes "let's just pipe it through ChatGPT and remove the names first."

The reframe: local AI is not a fun privacy project. It is risk management. If a cloud AI leaks one patient record, the liability is on us, not the vendor. The vendor's terms of service make sure of that.

"But anonymization?"

The natural objection: just strip the names before sending. Two problems.

First, anonymization is hard to do perfectly. Names are obvious; other identifiers (rare diagnoses, unusual procedure dates, distinctive case details) can re-identify a patient even with names removed. Studies have shown re-identification from "anonymized" data sets going back decades.

Second, getting it wrong once is the whole liability. The error budget is zero. Removing the cloud removes the risk class entirely. You cannot fail to anonymize if you never sent it.

Does this apply outside healthcare?

The PHI angle is healthcare-specific. The other two, intellectual property and tracking, apply to anyone with documents they would not paste into a public website. Internal architecture diagrams, unreleased product specs, draft contracts, financial models, anything subject to client confidentiality, anything subject to NDA. The threshold for "should this run locally" is lower than people assume.

FAQ

Is local AI as good as the cloud models? Not as raw-capable, no. GPT-4 and Claude Opus are bigger than anything you can run at home. But the demo later shows that for a focused task, read this passage and answer the question, a local model plus good retrieval gets you what you need. The capability gap matters less when the job is "read this and answer," not "write me a novel."

What about just anonymizing the data before sending it to the cloud? Anonymization is hard to do perfectly, and getting it wrong once is the whole liability. Removing the cloud removes the risk class entirely.

Does this apply to me if I am not in healthcare? The PHI angle is healthcare-specific. The IP and tracking arguments apply to anyone with documents they would not paste into a public website.

Isn't running models locally expensive? Less than people think. A used Mac Studio or a workstation GPU runs the models we use in this tutorial. The recurring cost, zero. The PHI breach you avoided, between half a million and a million dollars. The math gets generous quickly.

If you only remember one thing

Local was not a preference. It was a requirement. The data rules made the decision before any technical comparison was possible.