AI governance
A working pilot is not evidence of production readiness
AI systems fail silently rather than crashing, which is why a standard risk matrix mis-ranks them. A quiet hallucination in a citizen-facing service can outrank a visible outage that everyone noticed and fixed within the hour.
Executive summary
4
things to build in before scaling past the pilot
Core conclusions
- A proof of concept shows something can work under conditions you chose; production asks whether it keeps working under conditions you didn't.
- AI fails quietly — a plausible wrong answer with no error raised — which is why scoring it with FMEA (severity, occurrence, detection) reprioritises the register versus a standard risk matrix.
- A silent hallucination in a citizen-facing service can outrank a visible outage: the outage is detected instantly and recovered fast; the hallucination isn't.
A proof of concept demonstrates that something can work under conditions you chose. Production asks whether it keeps working under conditions you did not.
The gap between the two is where most AI programmes stall, and it is wider for AI than for conventional software because of how AI fails. Deterministic systems tend to fail loudly: a service stops, an alert fires, someone is paged. Probabilistic systems fail quietly, returning a plausible answer that is wrong, with no error raised anywhere.
Why standard risk matrices misprice it
A conventional matrix scores severity against likelihood. Scoring AI risk with FMEA instead — severity, occurrence and detection together — reprioritises the register, because detection is precisely where these systems are weak.
Run that scoring and a silent hallucination frequently ranks above a visible outage. The outage is severe but detected instantly and recovered quickly. The hallucination may be less severe per instance, occur steadily, and go undetected for months.
What to build in before scaling
- Detection as a first-class requirement: how would you know, and how quickly, if the system began being wrong?
- A defined and rehearsed rollback, with the fallback service specified.
- Sampling and review at a rate derived from the consequence rather than from convenience.
- An owner named for the system in month thirteen, which is when the original project team has moved on.
