Human Oversight
A human in the loop is a threshold, a queue, and a source of labels. Decide where the human decides, size the queue from the threshold, watch for automation bias, and remember that overrides are training data — and biased training data.
The problem, the obvious approach, and why it breaks
Every lesson starts where the work starts: someone has a problem, and the first model that comes to mind looks fine offline.
Where should a person make the decision instead of the model, and what happens to the model when the person's decisions become its next training set?
A content moderation team uses a model to remove posts. Removals are appealed, and the appeal team overturns a fraction. Product wants to automate more; trust-and-safety wants a human on every removal; nobody has said what a reviewer is actually for, or noticed that the reviewers' decisions are next quarter's labels.
Automate the confident cases and send the uncertain middle to a reviewer. Reviewers see the score to work faster. Their decisions go into the training set, so the model learns from human judgement and improves over time.
Reviewers agree with the model's suggestion most of the time, because the score is on the screen and the queue is long. The "human judgement" the model learns from is mostly its own previous output, and the training set converges on the model's errors (Feedback Loops).
- Reviewers agree with the model's suggestion most of the time, because the score is on the screen and the queue is long. The "human judgement" the model learns from is mostly its own previous output, and the training set converges on the model's errors (Feedback Loops).
- The review queue is sized by the threshold. A threshold change to reduce false positives triples the queue overnight, and the reviewers respond by rubber-stamping faster, which lowers label quality precisely when the model needs it most.
- Overrides — the cases where the reviewer disagreed — are the most informative labels and the rarest, and they are concentrated in the posts that reached review, which is a biased sample of everything the model scored.
- The high-confidence automated removals never see a human, so their error rate is unknown except through appeals, which measure who appeals rather than what was wrong.
What is being predicted, and from what data
This domain leads with these two. A target nobody defined precisely is a label nobody can trust, and a dataset nobody can describe is a model nobody can debug.
- Predict whether a post violates policy. The label is a moderator decision, which is a judgement made under time pressure with the model's score visible.
- The decision is remove, keep, or send to a reviewer. The review queue is the product of the threshold and the score distribution, not an independent design choice.
- One example is one post with text and image features, account history, and the moderator's decision. The decision was made by a reviewer who saw the model's score and the model's suggested action.
- Appeals produce a second label for a small, self-selected subset — posts whose authors bothered to appeal.
How it actually works
Precisely enough to predict its behaviour — not a framework API.
- A threshold on a probability creates three regions — automate-positive, automate-negative, and review. The review band's width is the queue size, and moving a threshold moves both the error rates and the workload at once (Threshold Selection).
- Automation bias is the tendency to accept a system's suggestion, stronger when the suggestion is visible, the reviewer is loaded, and the consequences of disagreement are effort. It converts the reviewer from an independent judge into an approver.
- When human decisions are labels, the label-generating process includes the model. The next model is trained on data whose distribution and whose errors were shaped by the current one, and labels from the review band are a non-random sample of the input space.
Where a person decides
The three regions of a threshold — automate-remove, automate-keep, review — are one design. Widening the review band buys human judgement on more cases and costs reviewer hours; narrowing it automates cases the model is least sure of. The queue is the product of the threshold, and any threshold discussion that does not include the queue is incomplete.
Not every review is the same review. A high-stakes decision needs a person because the error is irreversible; a low-confidence score needs one because the model does not know; a novel input needs one because the model has no basis. These are three routing rules, and the queue should say which one sent each item.
What routes a decision to a reviewer, and what is the reviewer for in each case?
when The action is irreversible or the harm of an error is severe regardless of the score — account termination, a medical flag.
cost Volume is driven by stakes, not by the model, so the model saves no reviewer time on these.
when The score lies in the band where the model's calibrated probability is near the threshold.
cost Band width is the queue; the reviewer sees the hardest cases, and their labels are the most valuable and the most biased by the score.
when Out-of-distribution detection or a new category the model has no training data for.
cost Requires a separate detector; false novelty alarms swamp the queue when the input distribution drifts legitimately.
when A sample of automated decisions, to measure the error rate where no human otherwise looks.
cost Reviewer time on mostly-correct decisions; the only measurement of the automated regions.
The reviewer sees the score
The offline evaluation of the moderation model was run on labels produced by reviewers who had the previous model's score on screen. The metric measures agreement with a process that included a model. It is not a lie, but it is a measurement of the loop, not of the world.
The gap shows up when a blind review sample is introduced: the model's agreement with blind reviewers is lower than with sighted ones, and the difference is the automation bias that has been inflating every metric since the first model shipped.
Agreement with reviewer decisions on the held-out set is high and has risen with each retrain.
A blind-review sample shows agreement noticeably lower, and appeals overturn a growing share of automated removals in a category the model was confident about.
- 1Sighted reviewers accepted the model's suggestion in most borderline cases, so the labels the model was evaluated against were partly its own output, and each retrain agreed with itself a little more.
- 2The category with rising appeals had been automated after the review band was narrowed, so no human had looked at it since, and its error rate was unmeasured rather than low.
- 3Policy for that category changed and the reviewers adapted; the model, trained on the old labels, did not (Concept Drift).
Overrides are labels
When a reviewer disagrees with the model, the disagreement is the most informative event in the system: it is a case where the model was wrong, identified by a person. It is also a label produced on a sample the model selected, by a reviewer who had to overcome the suggestion to produce it, under a policy that may have changed since training.
The training pipeline should know all of that. A label that records the model's suggestion, the human's decision and whether the score was visible can be weighted, audited and sliced; a label that records only the decision has laundered the model's influence into ground truth.
Reviewer decisions carry information beyond the model's score, so training on them can move the model rather than reinforce it.
holds when The blind-versus-sighted disagreement rate is stable and non-trivial; the queue is staffed so decisions are made in time to think; overrides are recorded as overrides.
breaks when The queue grows past what reviewers can judge and they approve to clear it; the score becomes more prominent in the tool; a policy change means reviewers and the model are applying different rules.
respond Restore the blind fraction, re-staff or move the threshold, and exclude the contaminated period's sighted labels from the next retrain rather than training on them.
1-- Every reviewer decision keeps what the reviewer saw. The training job decides2-- what to do with it; it does not get to pretend the model was not there.3CREATE TABLE review_labels (4 post_id text NOT NULL,5 model_version text NOT NULL,6 model_score numeric NOT NULL, -- the score at review time7 model_action text NOT NULL, -- what the model suggested8 score_visible boolean NOT NULL, -- FALSE for the blind sample9 reviewer_action text NOT NULL,10 is_override boolean GENERATED ALWAYS AS (reviewer_action <> model_action) STORED,11 seconds_spent integer NOT NULL,12 decided_at timestamptz NOT NULL13);14-- Automation bias, measured: agreement when the score was visible vs not.15SELECT score_visible, avg(CASE WHEN is_override THEN 0 ELSE 1 END) AS agreement16FROM review_labels WHERE decided_at > now() - interval '7 days'17GROUP BY score_visible;The two agreement numbers should be close. When the sighted one climbs and the blind one does not, the sighted labels have become the model's, and the next retrain will learn nothing from them.
How to build it
Most important first.
- Define what the human is for: high-stakes decisions where an error is irreversible, low-confidence scores where the model is genuinely uncertain, and novel inputs the model has never seen — each a different rule, each with its own routing.
- Size the queue from the threshold and staff it, or move the threshold and accept the error rates. Do not let the queue size be discovered on the day the threshold changes.
- Blind a fraction of reviews: the reviewer sees the post and not the score. Those decisions are the only labels that measure the reviewer rather than the model, and the disagreement rate between blind and sighted review measures automation bias.
- Sample the automated regions for human review too, at a low rate, so the error rate of the high-confidence decisions is measured rather than assumed (Evaluation Slices).
- Record the override as an override — the model's suggestion, the human's decision, whether the score was visible — so the training pipeline can weight or exclude it knowingly.
What to measure
Which number actually maps to the decision — and which numbers look relevant and are not.
- The override rate by score band, from blind reviews. This is the number that says whether the model is wrong in the review band, uncontaminated by the reviewer having seen the score.
- Queue depth and time-to-decision, as a function of the threshold. The relationship is the operational cost of any threshold move.
- Agreement between sighted and blind reviews on the same posts. A large gap is automation bias, and it means the sighted labels are the model's.
What must stay true after deployment
The field this whole domain exists for. A model is a set of assumptions with weights attached; these are the ones a monitor or a test should be checking.
- Reviewers are still exercising judgement rather than approving — measurable as a stable disagreement rate between sighted and blind review.
- The queue is staffed for the threshold in force, so labels are produced under a time budget that permits judgement.
- The training pipeline can distinguish an override from an agreement and knows whether the score was visible when the label was made.
- Offline: for the current threshold, compute the expected review volume from the score distribution before deploying, and compare with staffing.
- Online: the override rate by band from blind reviews, weekly; the sighted-versus-blind agreement rate; a sampled audit of automated decisions.
- Over time: track whether the model's errors are concentrating in the review band across retrains, which is the feedback loop closing.
What can go wrong
- The blind review sample is small and reviewers learn to recognise it, treating it as a test rather than a decision.
- Overrides are excluded from training to avoid bias, which removes exactly the cases where the model was wrong, so the model never learns them.
- The review band is narrowed to shrink the queue, the model automates the cases it is least sure of, and the error is discovered through appeals a month later.
- Blind review is slower and reviewers dislike it; the fraction has to be large enough to measure and small enough to staff.
- Sampling the automated regions for review is work spent on decisions that were probably right, and it is the only measurement of those regions.
- Recording override provenance makes the label pipeline more complex and the training set smaller if overrides are down-weighted.
- "We have a human in the loop, so errors are caught." A human who sees the score and has a queue is a rubber stamp with a name. Oversight is measured by the disagreement rate, not by the presence of a person.
- "Human labels are ground truth." They are decisions, made under conditions that included the model's suggestion, on a sample selected by the model. They are the best labels available and they are biased in a known direction.
- "Automate more to save reviewer time." Every point of threshold moved changes the error rate in the region that no longer gets a human, and that region's error rate is only known if it was sampled.
Where this applies
ML advice is stated as universal far more often than it is. These labels say what each claim is specific to — and where CONTESTED appears, the note gives the strongest form of the opposing view.
- DOMAIN-SPECIFICIn medical triage and credit the human is legally required and the design question is what they see; in content moderation the volume makes full review impossible and the question is which fraction; in fraud the reviewer is an investigator whose conclusion is the label by definition.
- CONTESTEDA serious position holds that showing reviewers the score is right: it makes them faster and more consistent, the automation bias is a feature when the model is better than the median reviewer, and blind review wastes the model's information. The counter-position is that a sighted reviewer produces labels that cannot improve the model beyond itself, so at least a measured fraction must be blind or the system stops learning. Both agree the disagreement rate must be measured; they differ on what fraction of the labels should be independent.
Where the depth lives
This domain teaches the model and hands the rest off by name.
- — Human factors — how interface design shapes automation bias, and how to staff a review queue, are operations and UX questions this lesson points at rather than answers.