Why an AI pilot stalls before production

The pilot worked. Everyone in the room saw it work. Six months later it’s still a pilot, the person who built it has moved on to something else, and nobody can say what is blocking it.

I’ve been called into that situation, and the cause was never the model. It was one of five things, and every one of them could have been seen before the pilot began.

1. Nobody agreed what “working” meant

A pilot with no threshold can’t fail, which means it can’t pass either. It produces a demo, the demo is impressive, and the decision to go further becomes a matter of mood.

Set the bar before you start: this many cases, this error rate, this cost per run, judged by this person. Write it down and don’t move it afterwards. When I run the proof stage of a diagnostic, the threshold is agreed before the experiment runs, for exactly this reason. A result measured against a bar chosen afterwards is a sales document.

2. The pilot ran on clean data

Pilot data is picked by somebody who wants the pilot to succeed. Production data is whatever turns up: the scanned form, the call recorded in a car, the customer who answers a different question from the one asked.

Run the pilot on a sample nobody tidied. If the result drops, you’ve learned the real size of the job while it was still cheap to learn.

3. There was no answer to “what if it’s wrong?”

This is the one that does damage. If being wrong is cheap and visible, such as a draft somebody reads before it goes out, ship it and move on. If being wrong is expensive and invisible, such as a figure in a report somebody acts on, then the checking is most of the engineering.

On a platform I built where the AI answers questions about money, every answer is reconciled against the ledger to a tolerance of one penny on value and 0.1% on rate. A run that reports a gain where the ledger shows a loss is a hard failure. That check took more work than the feature it protects. Without it the feature couldn’t have gone in front of a user, and no pilot would have revealed that, because a pilot has a person watching.

4. Nobody priced the running cost

Pilots run on a few hundred calls and a company card. Production runs on every record, every day, and per-call pricing that looked like nothing in the pilot becomes a line the finance director asks about.

Two things fix it. Log every model call with its token usage against a budget from the first day, so spend is a number you can read. And size the model to the task. A narrow job, classify this or extract these fields, wants the smallest model that clears the bar. In my own pipeline the local model is in production because it was run against a cloud model over identical input and judged good enough for that job.

5. It had no owner after the pilot

A pilot is a project. A production system is a responsibility: somebody has to notice when the model changes underneath it, when the inputs drift, when the exception queue grows. If that person wasn’t named at the start, the pilot ends in a meeting where everyone agrees it was promising.

Name the owner before you begin. If there’s nobody to name, that’s the finding, and it matters more than anything the pilot could tell you.

What to agree on day one

Before any pilot, get these onto one page:

• the business result it’s supposed to change, and by how much
• the threshold it has to pass, and who judges it
• the data it will run on, chosen by somebody with no stake in the outcome
• what happens when the answer is wrong, and who finds out
• what it will cost to run at full volume
• who owns it afterwards

If you can’t fill in a line, the pilot isn’t ready to start. Working out why you can’t fill it in is the more useful piece of work, and it’s the cheaper one.