Human oversight that actually oversees

Why "a human reviews it" isn't enough

Every system in Fernhill's inventory with real stakes has a person in the loop. The invoice reader proposes a total and due date; a clerk approves it before payment. The screening helper summarizes; HR decides. In Lessons 4 and 5, that human step was part of why these systems were judged acceptable.

But a person in the loop is only a control if they'd catch the model's mistakes. Decades of research on pilots, radiologists and operators of automated systems show the same pattern: when a system is usually right, people stop checking it closely. They agree with it faster, and they miss its errors more often. This is automation bias, and it gets worse with workload, time pressure, and a system that's right often enough to be trusted.

What the rules ask

The common thread: oversight has to be real. A reviewer who can't understand the output, has no time to check it, or never disagrees with it is not oversight under any of these.

Measuring it: plant known errors

Airport security faced the same problem decades ago: screeners look at thousands of bags, and real threats are rare, so how do you know they're paying attention? The US Transportation Security Administration's answer was threat image projection: the X-ray system occasionally projects a fictional threat onto a real bag, and records whether the screener spots it.

Fernhill does the same with invoices. About 4% of the queue are planted test invoices: real-looking invoices where the model's proposal is deliberately wrong (a total off by a digit, a due date a month early). The reviewer doesn't know which ones are planted. Separately, about 4% of real proposals are wrong too; Fernhill finds those later, when payments are reconciled.

Every review is logged with the week, the clerk, the kind of item, the decision and the seconds spent:

week,clerk,kind,decision,seconds,error_usd

The analysis computes, for each clerk:

def summarize(rows):
    n = len(rows)
    planted = [r for r in rows if r["kind"] == "planted"]
    errors = [r for r in rows if r["kind"] == "model_error"]
    correct = [r for r in rows if r["kind"] == "correct"]
    return dict(
        n=n,
        approve_rate=pct(sum(r["decision"] == "approve" for r in rows), n),
        median_secs=statistics.median(int(r["seconds"]) for r in rows),
        planted=(sum(r["decision"] == "reject" for r in planted), len(planted)),
        errors=(sum(r["decision"] == "reject" for r in errors), len(errors)),
        false_reject=pct(sum(r["decision"] == "reject" for r in correct), len(correct)),
        missed_usd=sum(float(r["error_usd"]) for r in errors if r["decision"] == "approve"),
    )

• approve_rate: the number most dashboards show, and the least useful.
• median_secs: the middle review time. Median, not mean, so a few long reviews don't hide many short ones.
• planted: planted errors caught, out of planted errors seen. Known right away.
• errors: real model errors caught, known only after reconciliation.
• false_reject: correct proposals rejected. Oversight can fail in this direction too.
• missed_usd: what the approved errors cost.

Four flags turn those into alerts:

FLAGS = [  # (test, threshold, message)
    ("planted_rate", lambda v: v < 0.80, "caught under 80% of planted errors"),
    ("median_secs", lambda v: v < 10,    "median review under 10 seconds"),
    ("approve_rate", lambda v: v > 0.99, "approves over 99%"),
    ("false_reject", lambda v: v > 0.05, "rejects over 5% of correct proposals"),
]

The thresholds are Fernhill's choices, not standards. Each says "look closer", not "this person failed".

The results

(Your machine)

python3 -I oversight.py reviews.csv

The script first simulates the log (seed 42, so it's the same every run), then analyzes it:

(simulated 6277 reviews into reviews.csv; seed 42)

== 1. Eight weeks, 6277 reviews

   clerk reviews approved median s  planted caught  real errors caught false rejects   missed $
   Ana      1200    92.8%       46     29/34 (85%)         44/53 (83%)          1.3%     10,016
   Ben      1877    95.8%       19     32/70 (46%)         31/72 (43%)          0.9%     49,293
   Cai      2400    99.1%        4       8/97 (8%)           5/93 (5%)          0.4%     96,590
   Dee       800    83.2%       70     35/37 (95%)         32/33 (97%)          9.2%        290

== 2. Flags
   Ana: none
   Ben: caught under 80% of planted errors
   Cai: caught under 80% of planted errors; median review under 10 seconds; approves over 99%
   Dee: rejects over 5% of correct proposals

Read the "approved" column on its own and nothing looks wrong: 83% to 99%, for a model that's right about 92% of the time (counting planted items). Now read the rest:

• Ana is what oversight should look like: 46 seconds a review, 85% of planted errors caught, few false rejects.
• Cai is a rubber stamp: 4 seconds a review, 8 of 97 planted errors caught. Cai reviewed more invoices than anyone, and let through the most errors: $96,590 paid wrongly.
• Dee is the opposite failure: catches almost everything, but rejects 9.2% of correct invoices. Each false reject means rework, a late payment or an annoyed supplier. Too little trust in a system is a cost too, and a sign that the reviewer doesn't understand when it's reliable.
• Ben looks average over eight weeks. Week by week, he isn't:

== 3. Ben, week by week
   week reviews median s  planted caught  real errors caught
      1     150       41             6/6                 5/6
      2     174       33             2/2                 6/7
      3     198       29             2/5                 5/9
      4     222       27            8/13                4/10
      5     247       22             4/8                3/10
      6     271       16            7/12                 2/7
      7     295       13            2/10                5/10
      8     320        8            1/14                1/13

Enlarge (opens the diagram full size)

Close

Ben's queue more than doubled, from 150 to 320 invoices a week. His time per invoice fell from 41 seconds to 8, and his planted catches from 6 of 6 to 1 of 14. Nothing about Ben changed; his workload did. That's the usual story behind automation bias in practice: not a careless person, but a review step sized for one volume and run at another.

Two cautions about reading this:

• Small weekly counts are noisy. Week 2's 2 of 2 and week 3's 2 of 5 are a handful of items each. Look at the trend and the review time together, not at one week.
• Planted catches tracked real catches. Over eight weeks, each clerk's planted rate was close to their real-error rate (Ana 85% and 83%, Cai 8% and 5%). That's what makes planted items useful: they give the same answer immediately that reconciliation gives months later. On a real log, check that this holds before relying on it.

== 4. What got through
   model errors: 251; approved anyway: 139 (55%); paid wrongly: $156,190
   approved errors by clerk: Ana 9, Ben 41, Cai 88, Dee 1

Over half the model's real errors were approved. A company that told an auditor "a human approves every invoice" would have been telling the truth, and it wouldn't have mattered.

Designing review that works

Measuring finds the problem. Design prevents it:

Spot the problem

Work out what's wrong before you read the explanation.

"Every decision is reviewed by a person." A lender's AI tool recommends approving or declining small business loans. Its documentation, its EU AI Act technical file, and its answer to a regulator all say "every decision is reviewed and confirmed by a credit officer." The dashboard shows officers agree with the tool 99.4% of the time, which management presents as evidence the tool is accurate. What's wrong? A 99.4% agreement rate is evidence of either an excellent tool or rubber-stamping, and the lender can't tell which. There are no planted cases, no review times, and no comparison with outcomes. Under Article 14, oversight must let people resist automation bias; under GDPR Article 22 and the SCHUFA ruling, a review that never changes anything may leave decisions "solely automated" in substance. Three fixes: plant known cases and measure how many officers catch; log review times and volumes and set limits; and report agreement alongside planted catch rate, so a high agreement rate can be read for what it is.

Where you are now

You can explain why a human in the loop isn't automatically oversight, measure it with planted items, read past approval rates, and design a review step that people can do well. Fernhill's screening helper and support bot would both be built on someone else's model. The next lesson is about assessing that vendor: what to ask, what to put in the contract, and what to check yourself.

Check yourself

Key terms

• Human oversight: people's ability to understand, monitor, override and stop an AI system, as a designed control.
• Automation bias: over-relying on a system's output, accepting it without enough checking.
• Rubber-stamping: approving without real review; oversight in name only.
• Planted item: a test case known to be wrong, mixed into real work to measure whether reviewers catch it.
• Threat image projection: the airport screening practice of projecting fictional threats onto real bags to measure screeners; the model for planted items.
• Override rate: how often reviewers change the system's output.
• False reject: rejecting a correct output; the opposite failure to rubber-stamping.
• Meaningful human review: review by someone with the competence, information, time and authority to change the outcome.

Go deeper

• The text of EU AI Act Article 14 (opens in a new tab), on human oversight.
• IAPP: key takeaways from the EU court's automated decision-making rulings (opens in a new tab), including SCHUFA.
• US GAO: screener training and performance measurement (opens in a new tab), on threat image projection.
• Try: change Ben's volume in the script to stay at 150 a week. What happens to his planted catch rate, and to the dollars missed?
• Try: if planted items were 1% of the queue instead of 4%, how many weeks would it take to notice Ben's decline?