How to take an AI pilot into production
A convincing prototype can still be unused three months later. This guide takes SME owners and operations leads through the stages from prototype to pilot to production, with the check to pass before each one and the integrations, review steps and ongoing support a tool needs in daily use.
A prototype tests value quickly, using sample data, a test user and the standard path through the workflow. Production is a different environment. It has to handle varied inputs, busy users and exceptions that never appeared in the demo.
The stages from prototype to pilot to production
The three stages answer different questions, and each has a check to pass before it starts.
- Prototype: a first version built around one task and a sample of representative data. It shows whether the output is good enough to be worth testing with the team. Before it starts, the trigger, inputs, outputs and approval points of the task are written down.
- Pilot: the intended users work with the tool on live cases for a limited period, typically two to six weeks. It shows whether they keep using it and whether the benefit can be measured. Before it starts, the criteria for moving on are agreed.
- Production: the tool is part of the daily process, with an owner, monitoring and support. Before it starts, the pilot has met its criteria and the four gaps described below are closed.
The prototype is also where a build-versus-buy decision is tested, because it shows at low cost whether a custom tool fits the work.
The four gaps between a demo and daily use
A pilot can stall when its owner changes role, when the tool stays outside the systems people use or when early errors go unresolved because nobody is assigned to fix them. Check four areas.
- Integration gap: the tool lives in a separate tab, away from the inbox, CRM or queue where the work happens.
- Exception gap: the standard path works, but unusual cases have no route to a person.
- Ownership gap: nobody is accountable for output quality, updates and support after launch.
- Trust gap: early errors were never triaged, so the team went back to the old way without saying so.
Each gap leaves a trace in the pilot. Usage that falls once the novelty has worn off points to an integration or trust gap. A growing pile of cases handled by hand points to an exception gap, and user questions that nobody answers point to an ownership gap.
Plan the production step before the pilot
Four decisions belong at the start of the pilot, while there is still time to design for them.
- Evidence: what the pilot must show, measured as time saved on the target workflow, the share of outputs accepted after review, the frequency of exceptions and whether users keep using the tool voluntarily.
- Home system: the system the tool will run in once it is in production.
- Owner: the person responsible after launch, with time set aside for the role.
- Budget: the estimated cost of the production step, so that a successful pilot is not left waiting for funds.
Write the thresholds down before the first user logs in. Suppose the pilot is a tool that drafts replies to support requests. The team might agree that it moves to production if most drafts are sent with only small edits, if exceptions remain a small share of requests and if the support staff are still using it in the final week without being asked.
Agree the stopping conditions in the same document. If acceptance stays low after several rounds of changes, if exceptions make up most of the volume or if the workflow itself turns out to be the problem, the right decision is to stop. A clear no is a useful result for a pilot.
Launch one workflow, then expand
Move one workflow for one team into production, with monitoring, a review step and a feedback channel. Block Consult builds custom AI tools the same way, around one defined task. Prove that the system holds up for a month, then extend it to neighbouring teams and workflows.
The review step needs its own design: who checks which outputs, against what standard, and what happens to the corrections. Our guide to human-in-the-loop review covers it.
Production checklist
Before extending the tool to more people, confirm five points.
- The tool is integrated in the system where the work happens.
- Exceptions are routed to a named person.
- Output quality is checked every week.
- A named owner has time set aside for the tool.
- The documentation is clear enough for the next hire to follow.
Budget for running and maintaining the system
Budget for logging, permissions, fallback paths, onboarding and documentation alongside the initial build. These items rarely appear in a demo, and the tool cannot carry daily work without them.
The cost continues after launch. Connected systems change, source documents go out of date and a model update can produce a different answer. Someone has to watch failures, output quality, usage and cost, and test changes against known cases before they are released.
That work can sit with your own team or with a partner under an ongoing AI care arrangement. Either way, include it when you calculate the return on the workflow, so the running cost is part of the decision to go into production.
Frequently asked questions
- What is the difference between a prototype and a pilot?
- A prototype tests whether the tool can produce useful output on representative data. A pilot tests whether the intended users get a measurable benefit from it in their daily work.
- How long should an AI pilot run?
- Two to six weeks is usually enough to produce evidence for a decision on a focused workflow. A pilot that runs much longer tends to signal that its question was unclear.
- What should a pilot measure?
- Time saved on the target workflow, the share of outputs accepted after review, how often exceptions occur and whether the intended users keep using the tool voluntarily.
- When is it wrong to move a pilot to production?
- When acceptance rates stay low despite iteration, when exceptions make up most of the volume or when the workflow itself turns out to be the problem. A clear no is also a useful pilot result.
Related reading
Let's build something durable.
Tell us about your project. We reply within two business days.

