How to assess a software agency when you cannot read the code

Write down what you think you own, then go and check it. Every check below works like that: a thing you believe about your own product, and somewhere you can look to see whether it’s true.

You’re paying an agency every month, the invoices arrive, the demos look fine, and you have a feeling you can’t put a name to. You can’t read the code, and none of the eight checks below needs you to.

1. Whose name is on the accounts?

Find out where the source code lives and who owns that account. Then the hosting, the domain, the app store listings and the database. If the answer to any of them is the agency, you’re renting your own product from them, and every other check on this list gets harder to act on because the party you’d be moving away from holds the keys to everything you’d be moving. Get the accounts into your company’s name before you raise anything else.

2. Can you see the work as it happens?

Get read access to the repository and the task board. You’re looking for a pulse: is work committed in small pieces through the week, or in one lump the day before the demo? Some good agencies run their boards in a way a client can’t read, so ask before you judge.

3. How does a change reach your customers?

Ask them to walk you through releasing a small fix. A healthy answer is short: reviewed, tested, staging, live, and it can be rolled back. An unhealthy answer involves one named person, an evening, and the word “carefully”.

4. What happens when it breaks?

Take the last outage and walk through it. How did they find out, how long did it take, what changed afterwards? If they found out from your customers, there’s no monitoring. If nothing changed afterwards, it’ll happen again.

5. Do estimates and invoices line up?

Take the last six pieces of work and compare what was estimated with what was billed. Everybody misses sometimes, me included. A team that misses in one direction every time either can’t estimate or has learned you’ll pay.

6. Who is doing the work?

You met a senior person in the pitch, so ask who has written your code this month. Agencies staff from a bench, and that’s normal so long as you’re told who’s on it.

7. Could somebody else take it over?

Ask for the documentation a new developer would need to get the system running on a fresh machine. If it exists, ask when it was last followed, and by whom.

8. Do they ever say no?

Did they ever tell you a feature was a bad idea, or that the cheaper route was to build nothing? An agency that builds whatever it’s asked is pleasant to work with. It’s also no help at all when the thing you asked for is wrong, and what you asked for is where the money goes.

What the answers mean

One or two weak answers is ordinary, and a conversation fixes it.

Weak answers on ownership, releases and handover together mean something structural, and no amount of chasing the account manager or rewriting the statement of work will fix a problem that lives in who holds the keys and who knows how to turn them. That’s the point to get an independent technical read of what’s been built: what state it’s in, what it would cost to move, and whether moving is the right call at all. Agree the days up front, and work out what to say to the agency so the build carries on while you decide.

Sometimes the agency is one supplier among several, with two freelancers and an automation platform beside it, and no one person can say which of them holds which password. That’s a question about the whole estate, and a software audit is where I’d start.