Running AI locally when the data cannot leave

The project gets as far as the data protection officer and stops. The recordings, the case files or the patient notes can’t be sent to a hosted model, and everybody concludes that AI is off the table.

It’s not. I know because that’s my own situation.

Why I run it this way

The business I run handles children’s data. We had thousands of recorded sales calls and onboarding notes that held the answer to a question we cared about, why leads weren’t converting, and sending those recordings to a model vendor was never going to happen.

So the sensitive stages run on hardware I own: speech-to-text, then a large open-weight language model. What crosses the boundary afterwards is structured conclusions and quoted evidence. No model vendor receives the recordings, and nothing trains on them.

That’s a working production system, not a workaround, and the pattern carries to any business with the same constraint.

The refusal is about one stage

When somebody says “our data can’t go to the cloud”, ask which data, at which step.

A pipeline has stages. Raw material comes in, gets transcribed or parsed, gets analysed, and produces a result. The raw material is the sensitive part. By the time you have a structured conclusion, such as “price was the objection in this call”, you’re holding something far less sensitive than the recording it came from.

So find the stage where the sensitive material is handled and run that stage locally. Everything downstream of it can use whatever tooling suits, because what reaches it has already been reduced. The constraint shapes one part of the architecture and leaves the rest alone.

What “local” can mean

There are three places the private stage can run, and the right one depends on your data and your people.

On your own hardware. The strongest position: the data never leaves the building. You need a machine with a capable graphics card and somebody to look after it.

On mine. For a business that doesn’t want to own the hardware, the same pipeline can run on equipment I control, under a contract that says what happens to the data. Whether that satisfies your data protection officer is their call, and it depends on the data.

In a private cloud. Your own tenancy, your own keys, no shared model endpoint. This suits a business that’s already in the cloud and whose objection is to a third-party model vendor, not to cloud hosting as such.

Working out which of those fits is part of the job, and it’s a conversation to have with whoever refused the first proposal, early.

Is a local model good enough?

This is the question everyone asks, and it has an empirical answer.

Open-weight models are smaller than the frontier models, and for a narrow, well-specified task that stops mattering. Transcribe this call. Pull these fields from this document. Decide whether this transcript mentions a price objection. None of that needs a model that can also pass a law exam.

Don’t take my word for it, or anybody’s. In my pipeline the same input runs through a local backend and a cloud one, and the two outputs are put side by side for a person to judge. The local model is in production because it was shown to be good enough for that job. Had it lost, it wouldn’t be there. Run the same comparison on a sample of your own material, using test data you’re allowed to send out, and you’ll have an answer you can show somebody.

What it costs

The economics run the opposite way to what people expect. A hosted model charges per call, for ever. Local hardware is a fixed cost you pay once, and after that the marginal cost of processing another thousand documents is electricity.

That changes what is worth doing. Reading an entire back catalogue is painful at per-call prices and ordinary on hardware you already own. For high-volume work on sensitive data, local is the cheaper route as well as the compliant one.

What to take to your data protection officer

Go back with a different question. Not “can we use AI on this data”, which has been refused, but this:

• the raw material is processed on hardware we control
• no third-party model receives it
• nothing is used for training
• only structured conclusions leave that stage, and here is an example of one

That’s a proposal somebody can assess. In my experience the first refusal was a refusal of the architecture they were shown, and nobody had shown them another.