Executive summary
4
requirements the new oversight design must meet
Key conclusions
- Human review becomes less reliable in the conditions where checking matters most: high volume, long duration and a low defect rate.
- Replacing a reviewer with a model does not transfer accountability. Both EU and Singapore regimes still expect a named person to answer for the outcome.
- A regulator asks whether the control layers are explainable, monitored and assigned to a named person.
Human review becomes less reliable in the conditions where checking matters most: high volume, long duration and a low base rate of defects. An automated checker does not get bored on the four-thousandth item. That is a real advantage, and organisations should use it.
The difficulty is that both EU and Singapore governance regimes assume a human is ultimately accountable for the outcome. Replacing the reviewer with a model does not transfer that accountability. The oversight design therefore has to be rebuilt around a person who supervises an automated checker instead of performing the check.
The new oversight design must meet four requirements
- Redefined human oversight: the person's job becomes sampling, setting thresholds and handling exceptions, and the skills the role requires change accordingly.
- Several independent layers of control, so that one automated checker is never both the reviewer and the only check on that reviewer.
- Full auditability of the checking layer itself, as well as of the system being checked.
- A maintained human fallback and override path, tested on a fixed schedule.
Regulators test whether each control is explainable, monitored and owned by a named person
A regulator asks whether the control layers are explainable, monitored and assigned to a named person. Whether they are automated matters less.
For smart city and infrastructure operations, this decides whether an oversight design holds up in an incident review, or whether the review finds that nobody was responsible.
