What does it really take to bring AI into production?
Make four decisions about a hypothetical AI project (scope, data, oversight, rollout) and see what you’d be trading away. A simulation, clearly labelled.
Engineering Desk · Engineering notes
In this story (5 sections)
General edition. , or use the switcher above.
Taking AI from prototype to production isn’t one decision, it’s four or five, made in an order nobody chose. Each one trades speed against something else.
This interactive essay lets you make those decisions for a hypothetical support-assistant project and explains the thinking behind the ratings it shows.
The short version
- Production AI is a sequence of decisions about scope, data readiness, human oversight and rollout. Each trades speed for reliability, governance or running effort.
- The explorer scores your choices on four ratings from 1 to 5. The ratings are editorial judgements for a hypothetical project and are not measured data.
- Narrow scope and staged rollout generally make failures cheaper to find and reverse. Human oversight is a running cost, not a launch task.
- Use it to start a conversation: make the four decisions in writing, and name who accepted each trade-off.
A hypothetical project, four decisions
Imagine an organisation that wants an AI assistant to help its support team draft replies. Nothing about it is real: no company, no data, no results. It exists so that the trade-offs can be seen without anyone’s actual project getting in the way.
Four decisions shape what happens next: how wide the scope is, how ready the data is, how much human oversight there is, and how the rollout proceeds. Each trades speed against reliability, governance or running effort. The explorer lets you make the four calls and see what you would likely be signing up for.
Make the four decisions
From prototype to production: an explorer
A static summary of the explorer. Four decisions, each with three options. Scope: one narrow task, three adjacent tasks, or a broad assistant. Data: use today’s data as it is, start from a small curated sample and widen it, or clean and document a core dataset first. Oversight: fully automatic, sample review with escalation, or a human reviewing every output. Rollout: launch to everyone, staged (internal, small group, everyone), or shadow mode first. In general, the narrow, staged, reviewed route is slower to launch and more reliable and governable; the broad, automatic, launch-to-all route is the reverse. All ratings are editorial judgements for a hypothetical project.
How the score works, and what it is not
Each option nudges four ratings up or down by one step from a neutral starting point of three: speed to first release, reliability, governance readiness and sustainability (how light the system is to run). The ratings are clamped between one and five and are shown with their working so you can disagree with any of them.
The model encodes a view that is widely held but not universal: that narrower scope and staged rollout usually make failures cheaper to find and reverse, and that human oversight is a standing cost rather than a one-off launch task. Continuous-delivery practice for machine learning, for example as described by Martin Fowler’s CD4ML write-up, leans the same way.
Where this model breaks down
Real projects have dependencies this model ignores: contracts, regulation, team skills, the cost of the model itself. A fully automatic launch can be sensible for low-stakes tasks, and a review-everything policy can be wasteful for trivial ones.
Treat the explorer as a conversation starter for a project team, not a decision tool.
What to take back to your own project
- Make the four decisions explicitly, in writing, before building starts.
- For each, name what you are trading away and who accepted it.
- Revisit oversight after launch: it is a running cost with an owner, not a checkbox.
- Frame the NIST AI RMF functions as questions for the team rather than paperwork.
Key takeaways
- 01Make the four decisions explicitly and in writing before you build.
- 02Every speed gain has a price in reliability, governance or running effort. Name it.
- 03Treat human oversight as a running cost with an owner.
Sources
- AI Risk Management Framework · NIST · accessed 1 Oct 2026
- Continuous Delivery for Machine Learning · martinfowler.com · accessed 1 Oct 2026
Your perspective
A better read starts with knowing what matters to you.
Three quick questions. A publication that adapts to your perspective.
Keep reading
All stories →Why promising AI pilots never become real products
A pilot proves something is possible. A product has to keep working. The gap between the two is predictable, and it can be closed before the money is spent.
A confidence score is not a probability
Many models emit a number between 0 and 1 that looks like a probability. Unless it has been calibrated, it isn’t one.
Designing experiences for millions of people
How product teams think about scale, usability and personalization when the rare case stops being rare.