Systems integration
Why the demo works and the production system does not
The gap is not the model. It is the operational architecture around it — context handling, evaluation, versioning, observability and cost control — none of which a demo is asked to have.
A prototype produces impressive outputs on inputs someone chose, at a volume nobody is paying for, with a person present to interpret anything odd. Production removes all three of those conditions at once.
What then surfaces is rarely a model deficiency. It is that the system has no disciplined answer for what context to include, what to do when the answer is uncertain, how to tell whether quality has moved, and what a bad day costs.
What production requires that a demo does not
- Context and memory tiering rather than a bigger window. Poorly managed context is what produces rambling answers, hallucinations and blown latency budgets.
- An evaluation set that predates the change you are about to make, so 'better' is a measurement rather than an impression.
- Versioning and shadow deployment, treating an AI release like a production software release.
- Observability at the interface boundary: what went in, what came back, how long it took, what it cost.
- A cost model that scales with adoption, because that is what usage does.
None of this is exotic. It is ordinary production engineering applied to a component that happens to be probabilistic — and it is the work that separates a system which holds under real traffic from one that degrades quietly while everyone assumes it is fine.