Running AI locally when the data cannot leave

The business I run handles children’s data, so the sensitive stages of what we built run on hardware I own. That setup exists because the first version of the proposal came back refused.

The project gets as far as the data protection officer and stops. The recordings or the case files or the patient notes can’t be sent to a hosted model, and everybody concludes that AI is off the table.

It isn’t. I’ve had the same conversation about my own business.

Why I run it this way

Speech-to-text first, then a large open-weight model, both running locally on machines I own. Only structured conclusions and quoted evidence leave that boundary, so no vendor holds the recordings and nothing trains on them. The open-weight model earned the job on a comparison: the same material through it and through a hosted one, the two outputs side by side, and had it lost it wouldn’t be there. The case study has the detail.

What the boundary costs me is hardware of my own, and the discipline of deciding for every new piece of the pipeline which side of the line it sits on. The deciding is the work; the hardware was the easy part.

The refusal is about one stage

When somebody says “our data can’t go to the cloud”, ask which data, at which step.

Raw material comes in, gets transcribed or parsed, gets analysed, and produces a result. The raw material is the sensitive part. By the time you have a structured conclusion, such as “price was the objection in this call”, you’re holding something far less sensitive than the recording it came from.

So run that stage locally, and everything downstream uses whatever tooling suits.

What “local” can mean

On your own hardware. The strongest position: the data never leaves the building, but you need a machine with a capable graphics card and somebody to look after it.

On mine. The same pipeline can run on equipment I control, under a contract that says what happens to the data.

In a private cloud. Your own tenancy, your own keys, no shared model endpoint, which suits a business already in the cloud whose objection is to a third-party model vendor.

Which of the three satisfies your data protection officer is their call, and worth asking before anybody designs anything. Have that conversation with whoever refused the first proposal, early.

Is a local model good enough?

Open-weight models are smaller than frontier models, and for a narrow, well-specified task that stops mattering. Transcribe this call. Pull these fields from this document. Decide whether this transcript mentions a price objection. None of that needs a frontier model.

Don’t take my word for it: run the same input through a local model and a cloud one, on a sample of your own material you’re allowed to send out, and put the two outputs side by side for a person to judge.

What it costs

A hosted model charges per call, for ever. Local hardware is a fixed cost you pay once, and after that the marginal cost of processing another thousand documents is electricity. For high-volume work on sensitive data, local is the cheaper route as well as the compliant one.

What to take to your data protection officer

“Can we use AI on this data” has been refused. Take this to the data protection officer instead, because every refusal I’ve seen was of one architecture and nobody had shown them another:

• the raw material is processed on hardware we control
• no third-party model receives it
• nothing is used for training
• only structured conclusions leave that stage, and here is an example of one