Human-in-the-loop approval that isn’t a rubber stamp

Most AI systems that act on money, customers or regulated records ship with the same safety plan: a person approves the output before it takes effect. It is a sound instinct, and in many deployments it quietly turns into a formality. The reviewer approves nearly everything, the errors that matter arrive inside the approved pile, and the organization can still say a human was in the loop.

This guide is about the gap between having a reviewer and having oversight. The research on automation bias is older than large language models, and its findings keep showing up in AI-assisted work. It also points at specific design levers: which decisions reach a person, what that person sees, how much they are asked to review in a shift, and what happens to their decisions afterwards.

How the loop becomes a rubber stamp

Human factors researchers have studied this failure for decades under two names. Automation complacency is the drop in checking that sets in when an automated system is usually right. Automation bias is the tendency to follow a decision aid instead of verifying it. Parasuraman and Manzey’s review of that literature found that complacency appears when the automated task competes with other work for attention, affects experienced operators as well as novices, and is not cured by simple practice. Automation bias produced both omission errors (missing a problem the aid did not flag) and commission errors (acting on advice that was wrong), and training or instructions did not prevent it.^[1]

AI agent products now report the same pattern with their own numbers. In August 2026, Anthropic said Claude Code users approve 97% of permission prompts, while they reject 39% of the plans Claude presents for approval.^[2] In a study it commissioned with 1,053 paid testers, one routine permission prompt per session was swapped for a clearly dangerous command. Testers blocked it 13.6% of the time, and the block rate fell from about 17% early in a session to about 5% after 50 or more earlier prompts.^[2] The company’s response was to make an automated classifier, rather than a prompt to the user, the default check on each action for its Pro, Max and Team plans.

None of this shows that people are bad at review. It shows that review is a task with a design, and a stream of near-identical approvals, almost all of them fine, trains people to click. A 2024 meta-analysis of more than 100 experiments points the same way: on average, human and AI combinations performed worse than the better of the two working alone. The losses were concentrated in decision tasks. Combinations gained when the humans were stronger than the AI on its own, and lost when the AI was stronger.^[3]

Regulators have written the problem into law. Article 14 of the EU AI Act requires high-risk systems to be designed so that the people assigned to oversee them can understand the system’s limits, stay aware of “the possible tendency of automatically relying or over-relying on the output” (the Act names it automation bias), interpret outputs correctly, override them, and stop the system.^[4] Article 26 requires deployers to assign oversight to people with “the necessary competence, training and authority, as well as the necessary support”.^[4] NIST’s AI Risk Management Framework asks organizations to define roles for human-AI configurations and to document their processes for human oversight.^[5] Neither text is satisfied by a button labeled Approve.

The question is not whether a human is in the loop. It is which decisions reach a person, what that person sees, how many they see an hour, and what happens to what they decide.

Route by confidence and consequence

The first lever is volume. If every output goes to review, the reviewer spends the day confirming correct work, and the rare error arrives as the ninety-ninth routine item of the hour. Machine learning has a name for letting a model decline to decide: selective classification, or the reject option. Geifman and El-Yaniv showed how to set a target risk level and have a classifier abstain on whatever inputs it must to meet it, trading coverage for accuracy.^[6] In an approval system, abstaining means sending the item to a person.

Two details decide whether this works. First, the score has to be calibrated: items scored 0.9 should turn out right about nine times in ten. A language model’s own statement of confidence is a weak starting point. In a benchmark of confidence elicitation methods, LLMs that verbalized their confidence tended to be overconfident.^[7] The reviewer decisions you already collect are the fix. Fit the mapping from raw score to observed approval rate for each kind of decision, and check it again whenever the model or prompt changes.

Second, confidence is not consequence. A payment to a changed bank account should reach a person even at 0.99, because the one error in a hundred costs far more than a misfiled support ticket. Article 14 asks for oversight measures “commensurate with the risks, level of autonomy and context of use”.^[4] So the router takes two inputs: a calibrated score, and a risk tier that policy sets for each kind of action. The model never sets its own tier.

The simulator makes the trade-off plain. A lower threshold sends more work to people and lets fewer errors past unseen; a higher one does the reverse. No setting is free. What it cannot show is that the reviewer’s catch rate is not a constant: the Anthropic data above suggests it falls as the queue grows. A threshold that floods the queue buys less safety than the arithmetic promises.

Here is the routing policy we start from. The numbers are placeholders; the structure is the point.

type RiskTier = "low" | "medium" | "high";
type Route = "auto_approve" | "review" | "escalate";

interface Proposal {
  kind: string; // e.g. "invoice.fee_waiver"
  rawScore: number; // model score in [0, 1]
  reversible: boolean;
}

// Set by policy for each kind of action and owned by the business. Never by the model.
const RISK_TIER: Record<string, RiskTier> = {
  "invoice.fee_waiver": "low",
  "customer.credit_limit": "medium",
  "supplier.bank_change": "high",
};

// Minimum calibrated score to skip review. null means a person always decides.
const AUTO_APPROVE_AT: Record<RiskTier, number | null> = {
  low: 0.95,
  medium: 0.99,
  high: null,
};

const GUESSING_BELOW = 0.6; // the model is guessing: send to a specialist, not the general queue
const AUDIT_RATE = 0.02; // share of auto-approvals re-reviewed blind

// Maps a raw score to the observed approval rate for this kind, fitted on past reviewer decisions.
type Calibrator = (kind: string, rawScore: number) => number;

export function route(
  p: Proposal,
  calibrate: Calibrator,
  rand: () => number = Math.random,
): { route: Route; tier: RiskTier; confidence: number; audit: boolean } {
  const tier = RISK_TIER[p.kind] ?? "high"; // unknown kinds are treated as high risk
  const confidence = calibrate(p.kind, p.rawScore);
  const bar = AUTO_APPROVE_AT[tier];

  let decision: Route = "review";
  if (confidence < GUESSING_BELOW || (tier === "high" && !p.reversible)) {
    decision = "escalate"; // specialist or two-person review
  } else if (bar !== null && confidence >= bar && p.reversible) {
    decision = "auto_approve";
  }

  const audit = decision === "auto_approve" && rand() < AUDIT_RATE;
  return { route: decision, tier, confidence, audit };
}

Note the audit flag on the auto-approve lane. It comes back in the section on feedback, because without it you cannot measure the errors the router lets through.

Show evidence, not a story

The second lever is what the reviewer sees. The instinct is to add an explanation, and the research says to be careful. Bansal and colleagues found that AI explanations raised the chance that people accepted the AI’s recommendation whether or not it was correct, without improving team accuracy.^[8] Steyvers and colleagues found that people overestimated the accuracy of LLM answers when shown default explanations, and that longer explanations raised their confidence even when the extra length added no accuracy.^[9]

The problem is the wrong kind of explanation, not explanation itself. Vasconcelos and colleagues argue that people choose strategically whether to engage with an explanation, weighing the effort of checking against the cost of trusting. Across five studies with 731 participants, explanations reduced overreliance when they made the AI’s answer cheaper to verify.^[10] That is the design target for an approval screen: checking has to be cheaper than trusting.

In practice the screen leads with the proposed action in one sentence. Below it sits a diff against the current state, then the specific source passages the proposal depends on, each one linked. The risk tier, and the policy that set it, sit next to the buttons. Uncertainty is stated in plain words and tied to a specific field. Kim and colleagues found that first-person uncertainty phrasing (“I’m not sure, but...”) lowered agreement with the system and raised accuracy, mostly by reducing reliance on wrong answers.^[11] Steyvers’ group found that aligning an explanation’s language with the model’s actual confidence narrowed the gap between what people believed and what the model knew.^[9] The model’s reasoning trace stays one click away, out of the default view.

For the highest tier, add friction on purpose. Buçinca, Malaya and Gajos tested cognitive forcing designs that make people engage before accepting, for example by asking for their own judgment first. Overreliance fell compared with plain explanations, but participants rated the most effective designs least favorably.^[12] That trade-off is why forcing functions belong on the small set of irreversible actions, such as the call-back check in the mock, rather than on every item in the queue.

Design the shift, not just the screen

Parasuraman and Manzey’s review includes a result that should shape every queue. In the early experiments, participants who only had to back up the automation, with no other duties, monitored it almost perfectly. When the automated task competed with manual work, detection of automation failures dropped, and it was far worse when the automation’s reliability stayed constant (33% of failures detected) than when it varied (82%).^[1] A reviewer approving AI output between other tasks, on a system that is right most of the time, is in the harder condition.

We plan review capacity the way we plan any operational role:

• A queue budget. Agree an hourly volume with the reviewers and let the router respect it. When volume spikes, the low-risk threshold moves; the reviewer’s pace does not.
• Batches of like items. Grouping by decision type lets attention go to what differs instead of re-reading context.
• Cheap, informative actions. Keyboard shortcuts for approve, edit and reject, and a reason code on every edit and reject, so a decision costs seconds and still carries information.
• Measured catch rates. Seed known-bad items at a low rate and score them separately, so attention is measured rather than assumed. It is the same method Anthropic’s study used on its testers.^[2]
• Short sessions. Rotate reviewers and cap session length, since catch rates in that study fell as sessions grew.^[2]

Every decision is a label

The last lever is what happens after the click. Each approve, edit and reject is a domain expert’s judgment on a real case: the data an evaluation set needs, and the hardest data to buy. The NIST framework makes the same point, noting that data about “the frequency and rationale with which humans overrule AI system output in deployed systems may be useful to collect and analyze”.^[5]

Active learning research adds a reason to value these labels in particular. A model can reach better accuracy with fewer labels when it gets to choose which examples are labeled.^[13] A confidence-routed queue is already that kind of chooser: the items that reach a person are the ones the system was least sure about, so their labels carry more information than a random sample would.

The same selection creates a blind spot. If people only see low-confidence items, nobody sees the errors the model made with high confidence, and those are the ones that reach production. The fix is a small random audit sample from the auto-approved lane, reviewed blind (without the model’s answer on screen) and stored with the queue decisions. Without it, the slip-through number in the simulator above is one you cannot measure.

Edits carry the most information, because the difference between the proposal and the reviewer’s version shows exactly what was wrong. We store every edit and reject as a candidate evaluation case, have a second reviewer confirm it before it counts, and run the suite before any model, prompt or threshold change ships. Calibration is then refit on the new decisions, which closes the loop.

kind: "supplier.bank_change"
raw_score: 0.94
sources: 3

Turning a decision into an evaluation case is a small function. What matters is provenance, so every case can be traced to the decision, the evidence and the model version behind it.

type ReviewerDecision = {
  proposalId: string;
  kind: string;
  input: unknown; // what the agent saw: request, records, retrieved passages
  proposed: Record<string, unknown>;
  action: "approve" | "edit" | "reject";
  final?: Record<string, unknown>; // the reviewer's version, for edits
  reason?: string; // reason code, required for edit and reject
  source: "queue" | "audit_sample";
  modelVersion: string;
  promptVersion: string;
  reviewer: string;
  decidedAt: string;
};

type EvalCase = {
  id: string;
  kind: string;
  input: unknown;
  expected:
    | { verdict: "act"; payload: Record<string, unknown> }
    | { verdict: "do_not_act"; reason: string };
  tags: string[];
  status: "candidate" | "confirmed"; // a second reviewer confirms before it gates releases
  provenance: Pick<
    ReviewerDecision,
    "proposalId" | "modelVersion" | "promptVersion" | "reviewer" | "decidedAt"
  >;
};

export function toEvalCase(d: ReviewerDecision): EvalCase | null {
  // Agreements from the queue are counted and sampled, not all stored as cases.
  if (d.action === "approve" && d.source === "queue") return null;

  const expected: EvalCase["expected"] =
    d.action === "reject"
      ? { verdict: "do_not_act", reason: d.reason ?? "unspecified" }
      : { verdict: "act", payload: d.final ?? d.proposed };

  const { proposalId, modelVersion, promptVersion, reviewer, decidedAt } = d;
  return {
    id: `eval-${proposalId}`,
    kind: d.kind,
    input: d.input,
    expected,
    tags: [d.action, d.source, ...(d.reason ? [d.reason] : [])],
    status: "candidate",
    provenance: { proposalId, modelVersion, promptVersion, reviewer, decidedAt },
  };
}

The primitives exist; the design is yours

Agent frameworks now ship approval as a built-in step, which removes the excuse that pausing an agent is hard. In LangGraph, a node calls interrupt() with a JSON payload. A checkpointer saves the graph state, the payload is returned to the caller, and the run resumes when you invoke the graph again with new Command({ resume }) on the same thread ID. The node restarts from its beginning on resume, so code before the interrupt must be safe to run twice.^[14]

import {
  Command,
  END,
  INTERRUPT,
  MemorySaver,
  START,
  StateGraph,
  StateSchema,
  interrupt,
  isInterrupted,
} from "@langchain/langgraph";
import * as z from "zod";

// saveReviewerDecision, applyAction and renderEvidencePanel are your own functions.

const Decision = z.discriminatedUnion("action", [
  z.object({ action: z.literal("approve") }),
  z.object({
    action: z.literal("edit"),
    patch: z.record(z.string(), z.unknown()),
    reason: z.string(),
  }),
  z.object({ action: z.literal("reject"), reason: z.string() }),
]);

const State = new StateSchema({
  proposalId: z.string(),
  kind: z.string(),
  payload: z.record(z.string(), z.unknown()),
  evidence: z.array(z.object({ ref: z.string(), quote: z.string() })),
  status: z.enum(["pending", "executed", "rejected"]),
});

const graph = new StateGraph(State)
  .addNode(
    "review",
    async (state) => {
      // Code above interrupt() runs again on resume, so keep it free of side effects.
      const decision = Decision.parse(
        interrupt({ kind: state.kind, payload: state.payload, evidence: state.evidence }),
      );
      // Runs once the reviewer has answered.
      // Key the write on proposalId so a retry cannot duplicate it.
      await saveReviewerDecision(state.proposalId, decision);

      if (decision.action === "reject") return new Command({ goto: "rejected" });
      const payload =
        decision.action === "edit" ? { ...state.payload, ...decision.patch } : state.payload;
      return new Command({ goto: "execute", update: { payload } });
    },
    { ends: ["execute", "rejected"] },
  )
  .addNode("execute", async (state) => {
    await applyAction(state.kind, state.payload);
    return { status: "executed" as const };
  })
  .addNode("rejected", () => ({ status: "rejected" as const }))
  .addEdge(START, "review")
  .addEdge("execute", END)
  .addEdge("rejected", END)
  .compile({ checkpointer: new MemorySaver() }); // use a durable checkpointer in production

// 1. Start: the run pauses at interrupt() and the payload comes back in __interrupt__.
const config = { configurable: { thread_id: proposal.proposalId } };
const paused = await graph.invoke({ ...proposal, status: "pending" }, config);
if (isInterrupted(paused)) renderEvidencePanel(paused[INTERRUPT][0]?.value);

// 2. Later, from the reviewer's authenticated session: resume the same thread.
await graph.invoke(new Command({ resume: { action: "reject", reason: "suspicious" } }), config);

The OpenAI Agents SDK works at the tool level. A tool marked with needsApproval (true, or an async function that decides per call) does not run until someone decides. Pending calls come back in result.interruptions, you call state.approve() or state.reject() on each, and you resume by running the agent again with that state, which can be serialized so a run can wait hours for a reviewer.^[15] Its documentation is also clear that approval state belongs on the server, and that the reviewer must be authenticated and authorized against the stored run rather than trusted from the request body.^[15]

These primitives give you the pause. They do not decide what should pause, what the reviewer sees, or what happens to the decision afterwards. A needsApproval function is a natural place to call a router like the one above, and the interrupt payload is where the evidence panel’s contents get assembled. Everything else in this guide is work the framework leaves to you.

A checklist

1. Route on a calibrated score and a risk tier that policy sets. Never let the model choose its own tier.
2. Keep queue volume inside a budget agreed with reviewers. Move thresholds, not people.
3. Lead with the action, a diff and the sources. State uncertainty in plain, first-person words. Keep the reasoning trace out of the default view.
4. Put forcing functions only on irreversible, high-risk actions.
5. Measure catch rates with seeded cases, and watch them across a session.
6. Store every edit and reject with a reason code, and audit a random slice of auto-approved work.
7. Gate model, prompt and threshold changes on the evaluation set those decisions build.

A human in the loop is a claim about a system, and whether it holds depends on engineering far from the model: the router, the screen, the shift and the log. It is work we plan from the first week of an agent build, because it is where oversight becomes real or turns into a checkbox.