On this page
Learning objectives
- Choose approval, review, sampling, and escalation controls
- Design review interfaces around changed fields and evidence
- Prevent human approval from becoming an unmeasured rubber stamp
Prerequisites
A task risk classification · Named process owner · Evaluation rubric
Answer first
Canonical definition
Human-in-the-loop design assigns a person a specific decision, review, labeling, or escalation role at a defined point in an AI workflow, with enough evidence and authority to change the outcome.
Operating context
Why it matters
Adding a human does not make a system safe if the review arrives too late, lacks context, or is expected to process an impossible volume.
The right control depends on impact, reversibility, confidence, and the cost of delay.
Method
Step by step
- 01
Classify decisions
Separate low-impact recommendations, reversible actions, external communications, regulated decisions, and irreversible changes.
- 02
Choose the control
Use pre-approval, post-review, sampling, dual control, or mandatory escalation based on risk.
- 03
Build the review packet
Show proposed action, changed fields, evidence, uncertainty, policy checks, cost, and available alternatives.
- 04
Measure reviewer behavior
Track approval rate, time, disagreement, corrections, missed errors, and signs of automation bias.
Hands-on lab
Design a high-signal approval step
Create a review screen for an agent-proposed external email or account change.
Deliverable
A decision rubric, evidence layout, approve/edit/reject paths, timeout behavior, and audit record.
What breaks
Common failure modes
- F1Sending every action to review until the queue becomes a bottleneck
- F2Showing only polished output without the evidence and changed state
- F3Treating an approval click as proof the reviewer understood the decision
Beyond the demo
Production notes
- Calibrate review load with sampling and risk tiers, and rotate reviewers where fatigue matters.
- Preserve the original proposal, reviewer edits, reason code, identity, and final executed action.
Further reading
Sources
Related Tenten resources