Executive summary
4
safeguards to build in before scaling beyond the pilot
Key conclusions
- The gap between pilot and production is wider for AI than for conventional software, because AI fails in a different way.
- AI fails quietly: it gives a plausible wrong answer and raises no error. Scoring it with failure mode and effects analysis (FMEA), which covers severity, occurrence and detection, therefore changes the order of the risk register.
- A silent hallucination in a citizen-facing service can rank above a visible outage. The outage is detected immediately and fixed quickly, while the hallucination is not.
A proof of concept shows that something can work under conditions you chose. Production tests whether it keeps working under conditions you did not choose.
Most AI programmes stall between these two stages. The gap is wider for AI than for conventional software because AI fails differently. Deterministic systems tend to fail visibly: a service stops, an alert fires and someone is paged. Probabilistic systems fail quietly. They return a plausible answer that is wrong, and no error is raised anywhere.
Standard risk matrices underrate silent AI failures
A conventional risk matrix scores severity against likelihood. FMEA scores severity, occurrence and detection together. Using it for AI risk changes the order of the risk register, because detection is exactly where these systems are weak.
Under that scoring, a silent hallucination often ranks above a visible outage. The outage is severe, but it is detected immediately and fixed quickly. Each hallucination may be less severe, but hallucinations occur steadily and can go undetected for months.
Build in four safeguards before you scale
- Detection as a core requirement: how would you know, and how quickly, if the system started giving wrong answers?
- A defined and rehearsed rollback, with the fallback service specified.
- Sampling and review at a rate set by the consequence of an error.
- A named owner for the system in month thirteen, when the original project team has moved on.
