Executive summary
5
capabilities that production needs and a demo never has to provide
Key conclusions
- A demo runs on chosen inputs, with no real volume, and with someone present to explain anything odd. Production removes all three conditions at once.
- In production, the usual gaps are missing context tiering, no evaluation baseline set before changes, unversioned releases and no observability at the interface.
- All of this is standard production engineering, applied to a component that happens to be probabilistic.
A prototype produces impressive outputs on inputs someone chose, at a volume nobody pays for, with a person present to explain anything odd. Production removes all three of those conditions at once.
It then becomes clear that the system has no disciplined way to decide what context to include or what to do when an answer is uncertain. It also has no way to tell whether quality has changed, or to show what a bad day costs.
Production requires five capabilities that a demo does not
- Context and memory tiering, before any move to a larger context window. Poorly managed context causes rambling answers, hallucinations and missed latency targets.
- An evaluation set built before the change you are about to make, so that 'better' is measured.
- Versioning and shadow deployment, so that each AI release is handled like a production software release.
- Observability at the interface boundary: what went in, what came back, how long it took and what it cost.
- A cost model that scales with adoption, because usage grows as adoption grows.
This is standard production engineering applied to a component that happens to be probabilistic. It decides whether a system holds up under real traffic, or degrades unnoticed while everyone assumes it is working.
