“Don’t worry, it runs in offline mode.” That sentence sounds reassuring. Usually it means someone found a setting called offline mode and switched it on. It does not always mean what it sounds like.

Here’s a real example. Open WebUI is one of the most popular chat front ends for running AI on your own servers, and it has an OFFLINE_MODE setting. Its own documentation says that setting turns off update checks and model downloads, but does not block connections to outside AI services.1 Nothing is wrong with the tool. The name just promises more than the setting does.

Other tools have similar defaults. Ollama, the simplest way to run a model on one machine, now ships cloud features that you switch off with a setting called OLLAMA_NO_CLOUD.2 vLLM, the engine many teams use to serve a model to lots of people at once, sends anonymous usage data by default until you turn it off.3 None of that is sinister. It’s just what happens when software is built for people who are online.

A setting is not a network

An air gap has a precise meaning. NIST defines it as an interface where systems are not physically connected, and any logical connection is manual and under human control.4 Not a checkbox. A missing wire.

Offline mode is a promise the software makes. An air gap is a fact about your network.

By that definition, the setups I build for small organizations that handle health information are not air-gapped either. The servers running the AI have no connection to the internet, and new models and patches are carried in by hand. But staff reach those servers from their everyday computers, which are online. That is an isolated server, not an air gap. It’s a strong control, and for my clients it was the right one. It just deserves its real name.

Why does the name matter? Because the name decides what people stop checking. If a board hears “air-gapped,” nobody asks where the prompts go. If it hears “isolated servers, reached from staff computers,” someone will reasonably ask what stops a staff computer from copying an answer somewhere else. That is the right question.

The three-question test

You don’t need to read a line of configuration to run this. Ask whoever runs your private AI, whether that’s your IT lead or a vendor, these three questions.

One: if we unplugged the internet from the machine running the model, what would break? The right answer names things: “model updates would stop, and we’d carry them in by hand.” The wrong answer is “nothing, it’s in offline mode.” If nobody has ever tried it, you don’t know.

Two: what outside addresses is that machine allowed to reach today? The right answer is a short list, or “none.” The wrong answer is a description of software settings instead of network rules. Settings live on the machine. Network rules live in front of it.

Three: who carried in the last model or security update, and when? If the answer is “it updates itself,” it is not isolated. If the answer is a name and a date, you have a real process. vLLM alone published 59 security advisories in 2026 through late September, so “we’ll get to it” is not a process.5

Anyone can ask these. An executive director can ask them in a ten-minute meeting. You’ll know you have a problem when the answers are about features and not about wires.

Most of you don’t need an air gap anyway

Here’s the part that may save you money. A true air gap is expensive to live with, because every model, patch and container image has to cross the gap by hand. Very few associations hold data that justifies that.

If your worry is vendors training on your members’ data, a contract may cover it. OpenAI says API data has not been used to train its models by default since March 2023.6 Anthropic says the same for its commercial products, and deletes API inputs and outputs within 30 days unless otherwise agreed.7 Even HIPAA, the rule my clients worried about most, does not require you to host AI yourself. It requires a business associate agreement with any vendor that handles protected health information for you.8

So the honest ladder has three rungs: a hosted service under the right contract, your own servers with no route to the internet, and a true air gap. Each costs more to run than the one before it. Pick the lowest rung that covers the data you actually have, and call it by its real name. I laid out the models, the costs and the trade-offs of all three in the longer piece on running your own AI. And if you’re on the first rung, W39 covers the vendor settings worth checking.

Quick takes

Open WebUI’s license changed. Since April 2025 it requires its branding to stay visible unless a deployment has 50 or fewer users in a 30-day period or buys an enterprise license.9 Unlimited internal users are fine if you keep the logo. Know that before you rebrand it as “our AI.”

Provenance reaches the plumbing. Texas’s January 2026 update to its list of technologies banned on state devices named BAAI, the maker of the widely used BGE embedding models, alongside Alibaba and other Chinese AI labs.10 The models that search your documents count too, not just the chat model.

“Zero retention” can depend on the model. Amazon Bedrock offers a zero-retention mode, but some of its newest models require a review mode that keeps prompts inside AWS for up to 30 days.11 Check the terms per model, not per platform.

Worth a read

The question I’d leave you with is simple. If someone asked your AI vendor to prove where the prompts went last Tuesday, could they show you a network rule, or only a setting?

Quick answers

Is offline mode in an AI chat tool the same as an air gap?

No. An offline-mode setting changes what one piece of software tries to do, and in Open WebUI it does not block connections to outside AI services. An air gap, as NIST defines it, means the systems are not physically connected and any transfer is manual.

Does HIPAA require an association to host AI on its own servers?

No. HIPAA requires a business associate agreement with any vendor that creates, receives, maintains or transmits protected health information on your behalf. Some organizations still choose their own servers to keep that information off outside systems entirely, but that is a risk decision, not a legal requirement.

How can a non-technical leader check whether our private AI is really private?

Ask three questions: what breaks if the model's machine loses internet, which outside addresses it can reach today, and who carried in the last update and when. Answers that name network rules, people and dates are good signs. Answers that only describe software settings are not.

From the Mind of Ravi Rooprai is a weekly column on association tech, data, and AI. Read the perspectives for the longer arguments behind it.

Researched with AI assistance and fact-checked against primary sources. The analysis, judgment, and writing are mine. How this column is made →