Measuring the Inputs to AI Governance

Most AI governance frameworks evaluate candidate workflows along a set of axes, each of which measures a certain source of risk. This post argues that functional AI risk governance should come with measurement protocols, to accurately estimate its inputs.

Consider a representative version of an AI risk framework that you AI capability team likely already has:

A classic approach to AI risk assessment is to score each workflow, delegate the low-risk tasks to AI, and use a case-by-case review for moderate-risk tasks to decide whether time saving is justified by governance cost. However, the risk frameworks assume that quantities such as detectability, reversibility, and time savings are treated as a given. In practical deployment, as shown in the last column of the table, these quantities are often subjective estimates produced by human managers and employees. Because psychological estimates can be biased in several ways, the accuracy of each estimate is itself uncertain.

To make an AI governance risk matrix work effectively with your deployment, you need one more tool: an input measurement protocol that treats the risk classifications as measurable quantities that we can continually track and estimate. The different ways how this can be done are discussed in the remainder of this post.

Specifiability (S)

Organizations often assess specifiability by asking whether a task can be described precisely enough for an AI system to perform it reliably. But the practical question is whether the specification captures the situations the system will actually encounter. The necessary benchmarks that evaluate prompts or agent workflows on a collection of representative examples are not sufficient, because they carry an unspoken qualifier: for examples like these. The difficulty arises when the deployment encounters genuinely novel situations. The widely discussed Moffatt v. Air Canada case illustrated this type of situation: an unusual interaction, underspecified in the model.

This suggests that specifiability should be viewed as a separate organizational KPI, measuring the ability to identify and integrate newly discovered edge cases. Instead of asking "Can we specify this task?", we recommend asking "How confident are we that our specification covers the situations we are likely to encounter, and how quickly can we incorporate the ones we missed?"

The quantity to be measured: S = P(new deployment falls inside current specification).

Error Detectability (D)

People routinely overestimate their ability to detect AI errors.

Ask a review team how likely they are to catch a bad AI output based on a given example, and you will get a confident answer because spotting an AI mistake is trivial once someone points it out. Finding it unprompted is a different task entirely, because most AI errors are embedded in otherwise fluent, well-formatted output that has no distinguishing surface feature — we call it plausibility masking.

In the vocabulary of visual attention research, this is conjunction search, not feature search. The target does not pop out, so detection requires slow, serial, effortful inspection of items that all look alike. Imagine finding a T among Ls in a large letter display. This is tiring, and performance decays steeply — a phenomenon known as the vigilance decrement, documented since Mackworth's radar-operator studies in the 1950s. Review quality is a declining function of queue position, even as employees remain engaged.

A related effect well documented in screening domains where targets are rare — in radiology, for instance — is that observers miss rare events at significantly higher rates than common ones. In Wolfe's low-prevalence studies, observers missed 7% of targets when targets appeared on half of all trials — and 30% of the same targets when they appeared on 1% of trials. The mental mechanism is a shift in the decision criterion, not a loss of sensitivity. When errors almost never appear, the reviewer's decision threshold rationally drifts toward "looks fine." Two decades of research using the Cognitive Reflection Test shows that even when a simple arithmetic mistake takes seconds to correct, most people miss it because they don't bother to check when the solution looks plausible.

This means that in a functional AI deployment where errors are rare and plausible-looking by design, your human miss rate will gradually drift up on the residual errors. As models improve, detectability estimates will likewise need to be continually updated.

The quantity to be measured: d = P(reviewer catches an error). This needs to be tracked longitudinally, and treated as model specific.

Reversibility (R)

Reversibility (R) measures how effectively an organization can recover after an AI-induced error. A workflow is reversible if the original outcome can be restored quickly, reliably, and at acceptable cost.

In practice, reversibility has three measurable components:

• Recovery cost: the resources required to correct an AI error — employee time, financial costs, reputational damage.
• Recovery success rate: the proportion of incidents in which the organization successfully restores the original outcome, tracked historically.
• Recovery latency: an AI error is only reversible if it is detected before the window for effective intervention closes. Measuring recovery latency therefore requires tracking both detection latency and the reversal window for each class of incident.

Impact (I)

Impact measures the consequences of an AI failure once it escapes recovery. In practice, losses incur through multiple channels — financial, liability, regulatory exposure, reputational damage, customer churn — so impact is best represented as a vector. Impact estimates should be derived from measurable organizational data whenever possible, such as historical incident costs.

Importantly, the expected loss from an AI deployment emerges from the multiplicative interaction between Detectability, Reversibility, and Impact, where expected loss is decomposed as:

This structure means that improvements to any one of Detectability, Reversibility, or Impact reduction compound: halving the miss rate and halving the recovery failure rate together reduce expected loss by 75%, not 50%.

Time saved (Δt)

Time saved by AI tends to be overestimated by the people saving it.

The most counter-intuitive evidence comes from METR's 2025 randomized trial of experienced open-source developers using AI coding tools on their own repositories. Before the study, developers expected AI to speed them up by roughly 24%. After completing the tasks, they estimated it had sped them up by about 20%. They were 19% slower with AI assistance, and they revised their initial expectation in the correct direction — they just did not revise it enough. There are various psychological mechanisms by which this can happen — prompt-writing overhead, or substitution of perceived effort for elapsed time — but the practical implication is independent of the mechanism: subjective estimates of productivity should not be treated as reliable measurements. Likewise, having people time-tracked in an obtrusive way will cause them to work differently as they are being observed. This means that objective, unobtrusive and ethical behavioural logging should be thought of as an integral part of AI governance infrastructure.

The quantity to be measured: Observed end-to-end task duration under contrasted AI availability (i.e. staggered rollout across teams).

Governance cost (C)

Governance cost measures the organizational resources required to use AI safely, across the new work created by oversight.

The quantity to be measured is governance cost per completed task, decomposed into four cost categories:

• Review: human oversight, escalation handling, and quality assurance.
• Infrastructure: API usage, monitoring systems, logging, storage, and evaluation infrastructure.
• Maintenance: prompt updates, model migration, and adapting workflows as AI models evolve.
• Compliance: documentation, audits, access controls, and regulatory reporting where required.

Implementing the measurement

The good news is pitfalls are avoidable once you recognize that the inputs to AI governance are measurable. Your practical implementation of a measurement framework can take several forms, depending on desired precision. For instance, here is the general implementation pattern, illustrated with D, in increasing order of ambition:

1. Seeded-error audits. The screening literature's answer to the low-prevalence problem is threat-image projection: inject known synthetic targets so that prevalence — and therefore the reviewer's criterion — stays level. The same approach works for AI oversight. Periodically inserting known-flawed AI outputs into the review stream yields three things at once: an empirical estimate of your true detection rate d, a counterweight to criterion drift, and a longitudinal per-reviewer skill metric — an early-warning system for deskilling.
2. Override calibration. Most organizations that log human oversight log the override rate. But the more informative KPI is the 2×2 design: correct acceptances, correct overrides, rubber-stamped errors, and spurious overrides of correct output. The two error cells have different fixes. Rubber-stamping calls for workload and prevalence interventions; spurious overrides call for trust calibration and better model documentation. Seeded errors give you ground truth for the rubber-stamp cell.
3. Risk-stratified review. Reviewing everything is infeasible, and the resource-rational oversight allocation should focus where P(error) × cost(error) is highest. In practice this means that an effective oversight strategy should stratify outputs by the D–R–I profile of their task class and sample review intensity accordingly. Part of this process can be automated, as even a weaker AI model is able to recognize when the stronger model's responce is perfectly safe, and when there is a chance that the the responce could have a different interpretation. Only these plausible failure cases need to be reviewed.

The equation to formalize the decision

Putting the pieces together, the expected net value of routing a task to AI integrates multiple variables. Here's one way we could think about this, assuming the following KPI are implemented:

Where Δt is the hours of human work the task no longer requires and w is the hourly cost of the person who would have done it — salary plus overhead, so the term Δt·w is the payroll value of the time saved. C_oversight is what review consumes (reviewer hours, escalation handling), C_infra what the system consumes (API, monitoring, maintenance); p_err is the model's error rate, d the probability a reviewer catches an error, and L the loss from an error that gets through.

Every parameter in this equation, including d, is measurable. Estimating these quantities accurately is the key to building an effective AI governance.

Building a system that learns and stays current

As a deployed AI workflow is a composition of (model_t, prompt, context), the system you are running will gradually diverge from the sytem you optimized. In practice, the models_t change with every provider update, and so the input-output mapping drifts. Most product teams invest into optimizing the prompts at launch, but under-invest in maintenance infrastructure that tracks model updates.

The fix is to recognize that prompt selection is not a one-time decision, and treat prompt variants as continued experimentation: retaining a shortlist of promising configurations, allocating a small fraction of live traffic to each, and letting performance data reweight them continuously. You may even find an older prompt variant outperforming the newer one after a model update, because the optimization was fit to a different model.

Where outputs are cheap to generate, ensemble or voting schemes across variants buy robustness against any single configuration deprecating. We will share more on this in upcoming posts.