Executive summary
5
steps for teams past the pilot stage, in order
Key conclusions
- A meaningful share of the work sent to a model is deterministic, such as a mapping, a threshold or a rules table. Sending it to a model adds nothing and incurs a charge for every call.
- Route each task by difficulty, add failover so that an outage moves traffic automatically, and manage governance centrally in one layer.
- Track cost per completed task. Ask any integrator, including us, which parts of the scope it recommends not automating yet.
For a while, the priority was speed: ship the pilot, connect the most advanced model and show the demo. That phase is ending. A more practical question about unit economics is replacing it.
A meaningful share of the work sent to a model is deterministic, such as a mapping, a threshold or a rules table. These tasks have a fixed cost, a predictable response time and an answer you can explain in one sentence. Sending them to a model adds nothing and incurs a charge for every call.
Teams past the pilot stage should take five steps
- Audit what is actually being sent to a model, and move anything deterministic back into code.
- Route by difficulty: use a small model or a rule for the common case, and the expensive model for the remaining cases.
- Build in failover, so that an outage or latency spike at one provider moves traffic automatically, with no manual action.
- Manage governance (rate limits, cost tracking and data controls) centrally in one layer, instead of separately for each integration.
- Track cost per completed task instead of cost per call, so that the figure reflects the work done.
This is one of the four questions we suggest asking any integrator, including us: which parts of this scope do you recommend we do not automate yet? An integrator that lists nothing has not examined the problem closely.
