A detector reaches 93% precision on common objects, so a team routes every proposal to annotators for quick correction. Reviewers soon discover that rare safety classes are missed entirely. Human-in-the-loop became human-after-the-model: people polished what the model found but did not search for what it ignored.
A real human-in-the-loop system defines where models propose, where people inspect independently, how rejection returns to correction, and how the evidence changes the next model or instruction set.
Human-in-the-loop annotation is often reduced to one sentence: let a model label first, then ask a person to fix it. That sequence is necessary but incomplete. A production system also needs routing, rejection, ownership, measurable correction, and a way to turn reviewer evidence into better models and better instructions.
The current Unitlab workflow builder supports Project, Annotate, Review, Model, Archive, and Complete stages on a connected canvas. Review has accepted and rejected paths. Ontologies, member assignment, workbench comments, project issues, and versioned releases provide the surrounding controls.
Operating principle: Humans must be able to discover model omissions—not merely edit model proposals. Otherwise the loop optimizes the visible candidates while rare, absent predictions remain invisible.
The workflow canvas as a control plane
A new live workflow began as Project → Annotate → Review → Complete. Stages can be added and connected, a project can be attached, ontologies selected, and a person assigned or work left open to anyone. The builder provides Save and Apply plus canvas navigation.
The Model stage makes inference part of that topology. Its settings cover the connected model, threshold, input type, queue scope, and class mappings. While automation owns an item, the Workbench is read-only; successful inference follows the configured route, and failures remain visible for recovery rather than being confused with a legitimate empty result.
Queues turn the graph into daily work
The Task Queue is organized by workflow stage. Each task exposes priority, media identity, derived status, assignee, and an Open action. Managers can select tasks and change priority, move them to another eligible stage, assign, or unassign them in bulk. Data Groups appear as one mixed work item rather than duplicating every tile.
The separate Batch Queue tracks upload and import processing with item count, progress, status, failures, data types, and assignee summary. Keeping these queues distinct prevents a stalled upload from being mistaken for a labeling backlog.
Four patterns for combining people and models
Model → Annotate → Review
The model creates proposals, an annotator corrects and completes them, and a reviewer checks the result. This is the safest starting pattern for a new integration because every proposal receives human attention.
Annotate with local AI assistance → Review
The annotator starts the task and chooses Magic Touch, Detect all objects, Find Similar, tracking, or interpolation when useful. This preserves human control over when assistance fits the item.
Model → Review with rejection to annotation
For a mature model on a narrow, measured distribution, a reviewer may accept clean results and reject difficult ones into correction. The review stage must still include a deliberate missed-object check.
Annotate → Review → Archive or Complete
Some data should not enter a model stage at all. Novel classes, ambiguous source material, sensitive judgment, or a manual gold set may need a fully human path.
One workflow is not automatically best for every class. Route according to measured error cost and uncertainty.
The rejected branch is where learning happens
Accepted labels increase volume. Rejected labels reveal the operating system. Require reviewers to classify why work was rejected: missing object, false object, wrong class, geometry, property, relation, temporal identity, instruction ambiguity, or source quality.
Annotation comments are suited to item-level context. Project issues carry status, responsible member, creator, and date, making them useful for recurring or blocking problems. Aggregate these signals by model version, class, source domain, and annotator cohort.
If the same error appears repeatedly, change the system: refine instructions, remap a model class, add a calibration example, adjust review routing, or curate the source distribution. Repeating the same correction is not a feedback loop.
Measure total accepted-unit economics
Proposal speed is not the target. Measure the complete cost of an accepted item:
- model execution and failure rate;
- annotator inspection and correction time;
- reviewer time;
- rejection and rework rate;
- missed-instance and false-proposal rates;
- class- and property-level error;
- queue waiting time; and
- final release throughput.
| Signal | What it can reveal |
|---|---|
| High correction, low rejection | Annotators are absorbing model error |
| Low correction, high missed-object errors | Review is focusing only on proposals |
| Errors concentrated by source | Distribution-specific weakness |
| Errors concentrated by class | Ontology, mapping, or model weakness |
| Long queue time, short work time | Assignment or workflow bottleneck |
Use a manual benchmark to calculate the actual lift. A model-assisted process that is 25% faster but creates more high-cost errors may be a poor trade.
Calibrate confidence only after observing outcomes
Confidence thresholds can route suggestions or control candidate volume, but raw scores are model-specific. Calibrate them against reviewed outcomes for the exact source population. Recalibrate when the model, ontology, or data domain changes.
Do not allow confidence to replace human uncertainty. Annotators need an honest escalation path when the source is ambiguous. An uncertain source and a low-confidence model are different conditions and may require different responses.
Separate gold data from assisted production
Maintain a manually created, independently reviewed benchmark that is not continuously rewritten by the same model being measured. It should include common and difficult cases, confusing negatives, rare classes, and source shifts.
Use it to compare model versions, prompts, thresholds, and mappings. Keep the benchmark tied to a specific ontology version. When definitions change, migrate deliberately or establish a new benchmark.
A worked routing example
Imagine a retail-image project with common product boxes and a high-risk restricted_item class. A model proposes both. Product boxes enter annotation and normal review; every restricted-item candidate and every image from a new store domain receives specialist review. Images where the model returns nothing still enter an annotator missed-object scan.
Reviewer corrections are categorized. The team discovers that most restricted-item error comes from one visually similar package and most product error comes from small shelf-edge objects. The next cycle adds confusing negatives to the benchmark, updates class instructions, and creates a curation slice for small objects. The model is not blamed abstractly; evidence changes data and policy.
After a stable period, clear common-product items may move from full review to sampled audit, while restricted items keep full review. The workflow becomes more efficient without pretending risk is uniform.
Design escalation and abstention
Humans and models both need permission to abstain. A model unavailable state should route to a human or a visible hold queue. An annotator uncertain about class meaning should comment or issue the case rather than guess. A reviewer should be able to reject due to source insufficiency, not only human error.
Define service levels for blocked work and an owner for each escalation. Otherwise, a careful annotator can be penalized by a growing queue of unresolved ambiguity.
Prevent automation bias
People tend to trust plausible proposals and spend less attention searching for omissions. Use calibration items with known model misses and false positives. Ask reviewers to inspect the source before focusing on proposals when risk is high.
Compare assisted and unassisted performance periodically. If annotator recall is lower with model proposals, the operating design needs a mandatory independent scan or a different routing policy.
Rotate some items into blind or manual audit so the team retains detection skill and the model remains measurable against an independent reference.
A day in the human-in-the-loop queue
At 09:00, a manager opens Task Queue and sees 800 items in Model, 120 ready for Annotate, and 46 waiting in Review. The numbers do not justify sending more data into inference: Review is already the constraint. The manager raises the priority of a rare-class calibration batch and leaves routine work in the shared pool.
When the model finishes one image, the task follows the configured edge into Annotate. The annotator opens it from the stage queue, corrects proposals, searches the source for missed objects, completes required properties, and saves. Save preserves annotation history but does not move the task. Send to Review performs the stage transition and advances to the next eligible item.
The reviewer rejects one item for a missing small object and accepts another. Rejection returns the first task to the configured Annotate stage with its reason and comment. The rework notification connects the decision to the person who can correct it. A malformed source is marked through the invalid path rather than counted as annotator failure; a model timeout remains a processing failure rather than a valid empty scene.
At midday, the manager selects several unassigned routine items and assigns them in bulk, then moves a deliberately held calibration item to specialist review. A grouped multimodal case appears once with its tile count and one assignee, not as several unrelated rows. Batch Queue is checked separately for two failed files from the morning upload.
At day’s end, the useful metric is not how many model predictions appeared. It is accepted units, correction time, rejection causes, queue age, failed automation, missed-object discovery, and recurrence of known errors. If review remains the bottleneck, the next action may be capacity, better routing, or lower upstream intake—not simply more inference.
Set service levels by stage and risk. Routine annotation may tolerate a longer queue than safety-critical review or rework returned by a reviewer. Priority changes should have an explicit reason; otherwise every stakeholder marks work urgent and the queue loses meaning. Monitor the oldest item as well as the average because a small difficult cohort can disappear inside healthy aggregate throughput.
Self-assignment can improve utilization when work is standardized. Restrict it when independent expertise, data boundaries, or customer commitments require named ownership. Manager override is useful for recovery, not as the default assignment method. The goal is a queue that makes responsibility visible while allowing the team to respond to real bottlenecks.
Follow one correction until it improves the next batch
Picture a morning queue of 500 product images. A model proposes boxes and classes, annotators correct them, and reviewers approve a sample. By noon the dashboard shows a recurring error: transparent bottles are often labeled as background, and green glass bottles are confused with plastic. A weak human-in-the-loop process fixes those images and keeps moving. A learning process traces the correction back to the system.
First, route work visibly. A Model stage creates candidates; Annotate lets operators accept, adjust, add misses, and complete required properties; Review accepts or rejects; rejected items return to a correction path. Model failure must also have an explicit path so items can continue manually or wait without appearing complete. The workflow should show ownership, not merely the sequence of screens.
At annotation time, capture the reason for meaningful corrections. A missing transparent bottle is a model miss. A green-glass/plastic confusion may be a class or property error. A disputed bottle_material value may expose insufficient evidence. A reviewer comment handles the one item; a project issue captures the repeated pattern with an owner and status.
Next, form a cohort. Filter Assets or project QA by class, property, source, model configuration, reviewer result, and error type. Inspect enough examples to determine whether the problem is local or systematic. If all errors come from one store's lighting, curation may need a source-balanced sample. If qualified people disagree on green glass, the ontology or instructions need a stronger example. If the label is clear but the model misses it, the correction should enter the next training-data version.
Do not immediately retrain on every correction. Protect a gold evaluation set and prevent duplicates or near-duplicates from crossing splits. Create a corrected dataset version, record the selection rules, and build a new release. The model team can then compare the next model against the same approved evaluation release while training on the new correction cohort.
Recalibrate automation only after outcomes are measured. A confidence threshold that routes transparent bottles to full review may be useful, but confidence is not the same as correctness. Compare error rate, correction cost, and reviewer burden by class and source. Raise automation where accepted-without-change performance is stable; keep humans closer to ambiguous or high-cost decisions.
Close the loop operationally. Update project instructions with the transparent-bottle example, add or refine the ontology option if necessary, record the issue resolution, publish the new dataset release, and note the model version that consumes it. The next batch should therefore differ in at least one observable way: better proposals, different routing, a clearer question, or better source coverage.
The key metric is not “percent AI-labeled.” Track reviewed cost per accepted object, escape rate after review, corrections by cause, repeated-error recurrence, time from issue creation to policy change, and model improvement on the protected evaluation release. Human effort becomes leverage when one correction changes the system that produced the next hundred items.
Prevent feedback from becoming a biased training set
Corrections are valuable, but they are not a random sample of production. Reviewers see more low-confidence, difficult, or high-risk items; easy accepted predictions may receive less inspection. Training only on corrected cases can therefore overrepresent failure modes without preserving the deployment distribution.
Keep provenance for assisted predictions, manual additions, reviewer changes, source cohort, and routing rule. When building the next dataset version, combine the correction cohort with representative production data and intentional negatives. Deduplicate against protected evaluation releases and keep case-level grouping intact.
Also separate human disagreement from model error. If two qualified reviewers choose different values under weak evidence, feeding one decision back as unquestioned truth may teach noise. Resolve the ontology, instruction, or uncertainty path first. Gold data should be calibrated independently of the current model's suggestions so the benchmark does not inherit automation bias.
The loop closes only after the new model is tested against a stable release and the result changes production policy. If performance improves only on the correction cohort, keep targeted review. If the improvement generalizes and reviewer correction falls, widen automation under a documented threshold.
Make the policy reversible. Record which model version, threshold, class, and source cohort receives automation, plus the condition that returns it to full review. A rise in repeated corrections, model failure, new source, or ontology change should automatically trigger reassessment rather than rely on memory.
Keep human workload visible. A threshold can reduce the number of items sent to Review while increasing the difficulty of every reviewed item. Track time, not only count, and ensure reviewers have enough context to challenge plausible predictions. Calibration and rest are quality controls when decisions are cognitively demanding.
The operating loop therefore has two clocks: fast correction for the current item and slower governance for model, schema, instructions, curation, and release. Both must complete for “human in the loop” to describe learning rather than perpetual cleanup.
Report those clocks separately. Item correction time shows production responsiveness; time from repeated error to deployed system change shows organizational learning. A team can be fast at the first and slow at the second, leaving reviewers to repair the same defect for weeks.
A weekly operating rhythm for the loop
Begin the week by reviewing model and queue health: invocation failures, accepted-without-change rate, correction by cause, reviewer rejection, Invalid items, queue age, and source or class shifts. Select one or two repeated error cohorts rather than reacting to every isolated correction.
Inspect examples in Unitlab. Compare model proposals, annotator changes, reviewer decisions, comments, issues, Item Properties, and source metadata. Decide whether the cause belongs to the model, ontology, instructions, curation, workflow, or insufficient evidence. Assign an owner and expected resolution.
Apply the smallest system change that addresses the cause. Add a negative example, refine a nested property, create an uncertainty state, adjust source selection, change a routing threshold, or prepare a corrected training cohort. Keep current-item corrections moving while the systemic change is tested.
Midweek, validate the change on a fixed calibration set and on new production examples. Review automation bias by comparing assisted and manual work. Protect gold and evaluation releases from contamination by current model suggestions.
At the end of the week, publish any approved data correction through a new dataset version and Release, update instructions or ontology versions, close or advance the issue, and record what the next model or batch should do differently. Do not close the issue merely because the original items were corrected.
In the following week, measure recurrence on a comparable cohort. If the error remains, revisit the diagnosis. If it falls without raising another severe error, consider changing review scope or automation. This cadence turns feedback into an owned sequence—observe, diagnose, change, validate, release, and monitor.
Try the workflow in Unitlab
Set a service level for every human stage. Task Queue priority changes which eligible item appears first, while assignment and self-assignment determine who can take responsibility. Measure queue age separately from active annotation time: a fast annotator cannot compensate for work that waits unclaimed for a day.
Managers should rehearse reassignment without losing accountability. The item history, comments, review result, and correction path must still explain who made each decision. That trace is especially important when a low-confidence model result moves through several specialists before acceptance.
Use rejected items as a structured learning sample: group them by correction reason, source slice, class, and model version before changing automation. This prevents one memorable failure from driving a broad threshold change that harms the rest of the data.
- In the workflow builder, map the current process as Project → Model or Annotate → Review → Complete.
- Add an explicit rejected path back to the accountable correction stage.
- Attach the project, ontology, and responsible people; define what happens when the model is unavailable or returns no result.
- Run a calibration batch with full review, including rare classes and intentional failure cases.
- Turn correction reasons into one owned change to the model, ontology, curation, or instructions, then verify it on the next batch.
Track missed-object discovery, correction effort, rejection cause, queue age, reviewer agreement, and recurrence of known errors. The loop is working only when the same failure becomes less likely to return.
The decision to make
Draw the rejected path before the happy path. Build a small Unitlab workflow and exercise model failure, human correction, and review rejection before scaling.
