Labeling and production feedback loops

At 3 a.m., your marketplace moderation dashboard looks excellent. The model reports 96 percent validation accuracy. It removes obvious weapon listings, leaves ordinary kitchenware alone, and its latency is fine.

Then support opens a different report: sellers are appealing removals at twice last month’s rate. Reviewers find that listings for antique swords, cosplay props, and collectible knives are being treated inconsistently. The model was not mysteriously possessed. Its training labels were inconsistent too.

One reviewer called an antique sword a prohibited weapon. Another called it a collectible. The training set contained both answers for nearly identical listings. The model learned the only rule available: guess.

You need a dependable way to produce labels, measure agreement, select examples for expensive human attention, and bring production evidence back into training without letting the model write its own exam.

That system is a labeling and production feedback loop.

A label is a specification, not a button

Suppose the marketplace model assigns one of two policy targets:

• remove: the listing violates the prohibited-item policy.
• keep: the listing may remain visible.

Those names do not define how reviewers should decide. A labeling guideline should specify:

1. The unit: the whole listing, an image, or the listing at a particular time.
2. Targets and statuses: remove and keep are targets; needs_review means the target cannot yet be determined.
3. The rule and evidence: what makes an item a violation, and which fields reviewers may use.
4. Edge cases and examples: antique swords, toy guns, replicas, parts, multi-item listings, positives, and near-misses.
5. Escalation and versioning: what happens when the policy does not settle a case, and which policy version produced the label.

For example:

Remove a listing when the item is a functioning weapon or a component whose primary purpose is to make one. Keep decorative replicas and toys. If the available evidence cannot establish the category, mark needs_review and send the case for adjudication.

After adjudication, the case receives remove or keep, or the policy changes. Do not create a third target class merely because an annotator was unsure. If the business genuinely needs three semantic outcomes, define those classes explicitly and collect adjudicated labels for all three.

Why ambiguity creates an accuracy ceiling

A model can only learn patterns that the data consistently connects to labels. Imagine 100 listings with the title “antique sword” and no useful image. Reviewers label 60 remove and 40 keep. A deterministic classifier seeing only those fields can choose the majority and get 60 correct; it cannot infer which reviewer was right.

If those examples are 100 of a 1,000-example test set and the other 900 are perfectly learnable, the best possible accuracy from this information is 96 percent: 900 clear cases plus 60 majority guesses. A larger model cannot remove this ceiling. A clearer policy, better evidence, or an escalation path might.

For a binary classifier, do not count unresolved needs_review cases as either target. Evaluate on adjudicated cases, then separately measure escalation coverage and the risk among non-escalated cases.

Measure agreement before trusting the labels

Inter-annotator agreement is a diagnostic for the labeling process, not a model score. Raw agreement can flatter you: if only 2 percent of listings are violations, two reviewers who always choose keep agree 98 percent of the time without useful discrimination.

Cohen’s kappa adjusts agreement for the agreement expected from each annotator’s label frequencies:

kappa = (observed agreement - chance agreement) / (1 - chance agreement)

Here is a worked example with 100 listings:

	B: remove	B: keep	A total
A: remove	30	10	40
A: keep	20	40	60
B total	50	50	100

They agree on 30 plus 40, so observed agreement is 0.70. Annotator A used remove 40 percent of the time; B used it 50 percent of the time. With those frequencies, chance agreement is:

• both remove: 0.40 × 0.50 = 0.20
• both keep: 0.60 × 0.50 = 0.30
• total: 0.50

Therefore:

kappa = (0.70 - 0.50) / (1 - 0.50) = 0.40

A kappa of 0.40 means modest agreement beyond the chance calculation. It is not 40 percent model accuracy. Kappa ranges from 1 for exact agreement to 0 for chance-level agreement and can be negative. With imbalanced labels, it may be low despite high raw agreement, so report the confusion table and class frequencies beside it. There is no universal “good” threshold; policy and error costs matter.

Measure independent decisions on a fixed, representative sample. If reviewers discuss every disagreement before recording answers, you are measuring post-adjudication agreement instead.

When reviewers disagree, adjudication should resolve the case and record why: an undefined edge case, missing evidence, or conflicting interpretations. Those reasons tell you whether to revise the guideline, improve the model input, or route similar cases to humans.

A small gold-standard set of expert-adjudicated ordinary cases and edge cases can calibrate reviewers and catch model regressions. Keep it stable, but do not use it as the only evaluation set; a model can memorize it.

Choosing which examples to label

Suppose 200,000 listings are unlabeled but you can afford 2,000 reviews. Random sampling provides an unbiased estimate of traffic, but may spend most of the budget on easy kitchenware. Active learning selects examples based on model uncertainty or dataset coverage.

For a binary model, a prediction of 0.51 for remove is less certain than 0.99. Margin sampling chooses the smallest gap between the top two class probabilities: 0.51 versus 0.49 has a margin of 0.02, while 0.98 versus 0.02 has a margin of 0.96. For multiclass models, uncertainty can use the top probability or entropy.

Uncertainty misses confidently wrong predictions in new regions. Diversity sampling selects representatives from different clusters of listing representations, helping expose new seller styles or image patterns. It may also spend labels on unusual examples that do not improve the target metric.

A practical queue mixes random examples, low-margin examples, diverse representatives, and targeted risk slices. Log the proportions so the resulting dataset remains interpretable.

Method	Chooses examples by	Best when	Main risk
Random	Uniform chance	Estimating real-traffic quality	Rare hard cases may be scarce
Uncertainty	Low confidence or high entropy	Finding boundary errors	Misses confidently wrong regions
Margin	Small top-class probability gap	A classifier has a clear boundary	Focuses narrowly near that boundary
Diversity	Coverage of data regions	New patterns may be missing	Coverage does not guarantee usefulness
Weak supervision	Rules, metadata, or existing systems	Millions of provisional labels are needed cheaply	Correlated, biased, noisy labels

The queue should also include gold items for reviewer checks and, after deployment, random audits of model decisions.

Active learning biases the resulting dataset. An uncertainty-heavy training pool can improve the boundary but misrepresent production traffic. Use it for training; use a representative, time-aware sample for evaluation.

Weak supervision: useful noise at scale

Weak supervision creates provisional labels from imperfect signals. A labeling function may be a rule, metadata field, or existing model that votes or abstains. Examples include:

• title contains “rifle” or “pistol” → likely remove;
• category is “toys” and title contains “foam” → likely keep;
• an existing moderation system removed the listing → likely remove.

These signals can conflict, such as “toy rifle.” A label model can combine overlapping votes and estimate source behavior under assumptions such as conditional independence given an unobserved target. But five rules copied from one keyword list may agree perfectly while sharing the same blind spot. Unlabeled agreement cannot generally reveal which source is correct.

Validate source quality and combined labels on a clean, human-adjudicated set. Treat weak labels as cheap proposals for human verification, not ground truth.

The production loop

A training set is historical. Production changes what gets seen and what gets labeled. Log:

• listing input and timestamp;
• model and policy versions;
• predicted probabilities and final decision;
• human overrides and appeals;
• label source and maturity.

The last field matters because outcomes may arrive weeks later.

Model-shaped and partial feedback

If the model removes suspicious listings before shoppers see them, shoppers cannot report listings they never saw. The next dataset contains fewer reports, making the model appear better while reducing the opportunity to observe its mistakes. Model outputs influence exposure, and that changed exposure generates future training data.

This is selection bias: observed outcomes differ systematically from the population you want to predict about. If only approved listings remain visible, “no report” among them is not proof of safety. Removed listings may be correct or false positives; their outcomes are missing because the decision prevented observation. This is partial feedback, not a negative label.

A chargeback may arrive 60 days after a transaction. If a weekly training job treats recent examples with no outcome as negative, it creates a systematically “clean” recent set. Track label maturity and states such as observed positive, observed negative, pending, not observable under this decision, and adjudicated by human review. Evaluate mature examples separately from provisional monitoring data.

A safer feedback loop

Combine learning with measurement:

1. Run representative random audits. Review some approved and removed listings, including confident decisions. With known coverage and mature adjudicated labels, this estimates decision quality in production.
2. Use governed exploration when needed. For a logged, safety-reviewed subset, vary exposure or routing—for example, send borderline listings to manual review instead of automatically blocking them. This helps observe outcomes under more than one action but has product and safety costs.
3. Record audit propensities. If an item has audit inclusion probability πᵢ, use a weight proportional to 1 / πᵢ for population metric estimates. This requires nonzero coverage for every target segment; weighting cannot recover a segment that is never audited.
4. Keep queues separate. Random audits estimate quality; uncertainty finds boundary errors; diversity finds new patterns; appeals diagnose failures; gold items monitor reviewers.
5. Evaluate through time. A later time window exposes policy, seller, and behavior changes that a random split can hide. Monitor drift.
6. Version provenance. Record who labeled an example, under which guideline, from which queue, at what maturity, and whether the label was observed or inferred.

The fundamental limitation is that labels are measurements made by people and systems under a policy, not reality itself. A sampler cannot repair a policy experts cannot apply consistently, and a weak-label model cannot recover outcomes the product never allowed you to observe. Sometimes the right improvement is a clearer policy, human escalation, or a product change—not a larger classifier.

For this marketplace, keep needs_review as a status unless the policy genuinely defines it as a semantic class. Evaluate binary predictions only on established remove and keep targets, and report escalation coverage and non-escalated error separately. If three semantic outcomes are required, define a three-class task and collect adjudicated three-class labels.

Quick check

0/3

Q1

0.200.400.500.70

Check answer

Q2

Uncertain examples cannot be labeled by humansThe queue contains examples selected for model uncertainty rather than the production distributionUncertainty sampling always removes rare examplesEvaluation sets must contain only gold-standard examples

Check answer

Q3

Increase the model size so it can learn the phrase betterLabel only the listings with the lowest confidenceCreate a random audit or governed exploration sample, keep pending and unobservable outcomes distinct, and review blocked listingsDelete the phrase from the training data

Check answer