Vision Datasets: Label Rules and Distribution Shift

Vision Dataset Labeling and Distribution Shift

Vision dataset quality depends on what labels mean and which images the evaluation actually represents. Start with written labeling rules, then evaluate the conditions where the model will be used: image quality, lighting, viewpoint, environment and consequential uncommon cases.

Separate a label problem from a distribution problem. An unclear rule can make the reference answer inconsistent; a consistent label set can still omit the conditions encountered later. A single aggregate score cannot resolve either issue by itself.

TL;DR

Write inclusion, exclusion and ambiguity rules with positive and negative examples.
Keep image source, annotation version and review history traceable.
Prevent related images or acquisition groups from leaking across intended evaluation splits.
Report slice counts and task-appropriate errors alongside aggregate metrics.
Treat missing or tiny slices as uncertainty or coverage gaps, not proof of robustness.

Quick Example

This complete Python fixture checks fictional binary-classification outcomes. It compares day and night groups and retains an absent fog group as None, rather than reporting perfect accuracy over zero images. There are no image files, annotations from people or model predictions behind these invented rows.

Expected output confirms slice arithmetic and reports missing fog coverage. The day/night values of 1.0 and 0.5 are teaching arithmetic over two rows each, not measured model performance or statistically reliable comparisons. The checks do not validate a labeling process, trained model, deployment population or vision service.

Core Concepts

Define the annotation task

Classification, object detection and segmentation ask annotators for different outputs. A class label, bounding box and pixel mask are not interchangeable. Google’s labeling overview recommends explicit definitions, edge cases, annotator training and quality checks. Use that as process guidance, not a guarantee that human or automated labels are correct. Data labeling

Match image conditions to intended use

For its AutoML image workflows, Google recommends data resembling expected prediction images, including relevant blur, resolution, angles and backgrounds. The general lesson is to inspect the intended operating conditions; its product-specific sample-count guidance is not a universal requirement for every vision task. Image data preparation

Distribution shift is an evaluation boundary

The WILDS research studies naturally occurring changes across settings such as camera locations, institutions and time. It demonstrates why in-distribution evaluation and performance under a different distribution are separate questions. Its results do not establish the behavior of an untested application. WILDS paper

Write rules an annotator can apply consistently

Define the unit of annotation and the observable evidence required for each label. For a fictional box/not-box image classifier, decide whether a partially visible package counts, how reflections are handled, and whether an image with multiple objects can receive one label or several.

Use a small calibration set with disagreements discussed before scaling annotation. Retain unresolved cases and route them to an appropriate adjudicator. Agreement can reveal consistency, but two annotators can agree on a shared misunderstanding; compare against the task definition and consequential examples too.

Separate “not present” from “cannot determine” when that distinction matters. Forcing every unreadable image into the negative class can teach the model the wrong relationship. Decide how uncertain or excluded examples affect training, evaluation and the deployed fallback.

Record rule revisions with annotation provenance. If a definition changes, identify affected labels and evaluation versions before comparing scores. An apparent model improvement can result from changing the reference labels rather than better predictions.

Evaluate conditions without hiding denominators

Choose slices from the expected operating conditions and failure consequences, then preserve their counts. Useful dimensions may include day/night, blur, resolution, camera/site, viewpoint, background, weather, object size and occlusion. Use only metadata appropriate to the task and data permissions.

Report task-appropriate errors. For a detector, missed objects and false detections at a declared matching/threshold policy matter; a generic classification accuracy does not replace detection evaluation. For a classifier, inspect confusion and per-class behavior as well as overall accuracy. State thresholds and label/split versions with the result.

Keep absent and small groups visible. Two night images cannot support a strong deployment conclusion, even if both are correct. Use an uncertainty method appropriate to the metric and sample design where needed, and collect representative additional examples before making consequential acceptance claims.

Control related-image leakage. Near-duplicate frames, repeated scenes, the same physical object or the same acquisition session can make a random image split easier than the intended deployment setting. Define the split unit from the generalization question and keep related groups together where required.

Avoid repeatedly tuning against the final test set. Use development data for iteration and maintain an evaluation set or prospective procedure appropriate to the decision. When a failure is added to training, preserve a separate way to test whether the change generalizes beyond that exact image.

Inspect errors with their context. A performance gap can reflect label quality, missing conditions, preprocessing, threshold choice or model limitations. Verify the cause before assuming that augmentation or more training will repair it. Synthetic transformations can help probe a hypothesis, but they do not automatically reproduce the full real-world distribution.

Best Practices

Preserve reference meaning

Keep label definitions, ambiguity rules and annotation versions with evaluation records. Review changed labels before interpreting score changes.

Report coverage alongside errors

Show counts, missing conditions and uncertainty. A blank slice is a collection task, not a successful test.

Challenge the intended deployment boundary

Evaluate representative acquisition groups and real conditions. Use a useful fallback or narrow the supported scope when evidence remains inadequate.

Common Mistakes

Bad: Turning every unreadable image into a negative label.

Correct: Use the declared ambiguity/deferral policy and preserve why the answer is uncertain.

Bad: Reporting an empty slice as 100% accurate.

Correct: Retain its zero count and undefined metric, then record the coverage gap.

Bad: Randomly splitting adjacent video frames across train and test.

Correct: Choose grouping that prevents leakage for the actual generalization question.

FAQ

Does high annotator agreement prove the labels are right?

No. It demonstrates a form of consistency under the measured process. Shared misunderstandings and incorrect task definitions still require review.

Is low-light accuracy enough to establish robustness?

No. Report its count, uncertainty, image sources and other relevant conditions. Real deployment behavior can differ along untested dimensions.

Can augmentation replace collecting real failures?

It can probe or enrich specific conditions, but synthetic changes do not guarantee representative coverage. Validate improvements on suitable independent real examples.