Why an AI pilot stalls before production
The pilot worked and nothing shipped. Five reasons that happens, all of them visible before the pilot started, and what to agree on day one so yours is not the next.
The model was never the problem. In every stalled pilot I’ve been called into the cause was one of five things, and all five were visible before the pilot started.
The shape is always the same. The pilot worked, everyone in the room saw it work, and six months later it’s still a pilot, the person who built it has moved on, and nobody can say what’s blocking it.
1. Nobody agreed what “working” meant
A pilot with no threshold can’t fail, so it can’t pass either, and whether it goes further comes down to mood. Set the bar before you start: this many cases, this error rate, this cost per run, judged by this person. A bar set after you’ve seen the result isn’t a bar. When I run the proof stage of a diagnostic the threshold is agreed before the experiment runs, and if you need a week of output to know what to measure, take the week, then set the bar and don’t move it.
2. The pilot ran on clean data
Pilot data is picked by somebody who wants the pilot to succeed. Production data is whatever turns up: the scanned form, the call somebody took in a car.
Run the pilot on a sample nobody tidied, and if the result drops you’ve learned the real size of the job while it was cheap to learn.
3. There was no answer to “what if it’s wrong?”
Cheap and visible, ship it and fix what turns up. Expensive and invisible, and checking the answer becomes most of the engineering. Where AI automation pays has the test I run before a build starts. On one platform I built, no answer about money reaches a user until it’s been checked against the ledger, and that check took more work than the feature it protects.
4. Nobody priced the running cost
Pilots run on a few hundred calls and a company card. Production runs on every record, every day, and per-call pricing that looked like nothing becomes a line the finance director asks about.
Log every model call with its token usage against a budget from day one. And size the model to the task, because a narrow job doesn’t need a frontier model.
5. It had no owner after the pilot
A production system is somebody’s responsibility: someone has to notice when the model changes underneath it, when the inputs drift, when the exception queue grows. Name that person before you begin, and if there’s nobody to name that’s the first thing to fix, because otherwise the pilot ends in a meeting where everyone agrees it was promising and nobody picks it up.
What to agree on day one
Six lines, and a blank one means the pilot isn’t ready to start.
- the business result it’s supposed to change, and by how much
- the threshold it has to pass, and who judges it
- the data it will run on, chosen by somebody with no stake in the outcome
- what happens when the answer is wrong, and who finds out
- what it will cost to run at full volume
- who owns it afterwards
Filling those in takes a morning, and it’s the same list I work through before a proof stage of a diagnostic runs.