Versioning Training Data in Unitlab: Reproducible Releases for ML Teams

Model B beats Model A by four points. The code diff is small, but the training folder was refreshed overnight and the ontology renamed two classes. The improvement may be real, accidental, or incomparable. Without a frozen data reference, the experiment has no defensible cause.

Training-data versioning turns a moving annotation operation into a controlled model dependency. A useful release records not only files and labels, but the schema, quality gate, selection logic, limitations, and relationship to prior versions.

Code is versioned by default. Training data often is not. A folder changes, an annotator corrects labels, a schema is renamed, and the next model run quietly trains on a different reality. When performance shifts, the team cannot tell whether code, parameters, source distribution, or label meaning caused it.

Unitlab datasets display versions, and releases provide a dedicated version selector, sample data, structured ground truth and metadata, Clone Release, and Download. These controls provide the foundation for data release engineering: turning an approved training state into an immutable experimental dependency.

Reader takeaway: A reproducible model run needs an immutable data reference plus the schema, selection logic, quality gate, and known limitations that gave that snapshot meaning.

What belongs in a data version

A useful version covers more than files and labels. Record:

• source locations and acquisition window;
• asset-selection and exclusion criteria;
• dataset version and split assignment;
• ontology ID and version;
• project and workflow configuration;
• model, prompt, or tracking assistance used;
• annotator and reviewer policy;
• quality metrics and unresolved limitations;
• release ID and version; and
• downstream experiment IDs.

This lineage lets a team reproduce the interpretation of the data, not merely retrieve the bytes.

Define release significance

Use a version convention that communicates impact. The exact numbering scheme matters less than a shared rule.

Consider a major release when label meaning, split methodology, parent grouping, or intended task changes. Consider a minor release when new approved data or a meaningful correction batch is added under stable definitions. Use patch-level increments for small corrections that do not change the contract—if the organization needs that granularity.

Do not use a version bump to hide an incompatible purpose. A detection dataset becoming a segmentation corpus should usually become a new product lineage.

Create a release manifest

The Unitlab release object exposes data, ground truth, metadata, visibility, date, count, version, and downstream actions. Add an organizational manifest that explains context.

Store the manifest beside the downstream training configuration or in the experiment system, with the Unitlab release reference as the canonical data checkpoint.

Make release creation an approval gate

Before release, validate counts, modality distribution, required properties, unresolved workflow states, rejected items, duplicates, split leakage, and representative previews. Confirm that schema-deletion warnings have been resolved deliberately.

The release owner should sign off on known limitations and intended use. For sensitive or regulated programs, add the required domain, security, privacy, or quality approvals.

Link every model run to one release

A training configuration should reference the Unitlab release ID and version. Record the selected UUEF, Standard Bundle, or compatible family-specific export, the downloader or loader version, and any transformations applied after retrieval. Validate export structure and preprocessing with the consumer.

The input side is versioned separately. Dataset edits remain in a mutable working draft until Publish version creates an immutable DatasetVersion. A project attaches the latest or an exact published version, so later changes to the source draft do not silently alter in-flight project data. A new source version must be published and attached deliberately.

If a transformation filters records, converts geometry, or remaps classes, version it as part of the data contract. “We trained on release v3” is insufficient if two teams preprocess v3 differently.

Evaluation data needs even stronger stability. Do not silently correct evaluation labels during a comparison. Publish a new evaluation release and rerun the relevant baselines.

Use cloning as controlled branching

Clone Release can seed a new project or experiment from an approved state. Treat the clone as a branch with parent, purpose, owner, expected differences, and merge or retirement plan.

Examples include testing a revised ontology, creating a customer-specific subset, adding hard negatives, or applying a new model-assisted workflow. Compare the branch to its parent on a stable benchmark.

Handle corrections without rewriting history

When a label error is found, record its class, severity, and affected releases. Correct the working data, complete review, and publish a new release. If the error could materially affect models already trained, identify consumers and decide whether retraining or reevaluation is required.

Unitlab's deleted-ontology-item warning is an especially important trigger. Restoring or keeping a removed schema item changes how historical annotations resolve. Treat the decision as a release event.

A reproducibility drill

Periodically ask a different team member to reconstruct an older experiment using only recorded references. They should be able to retrieve the release, ontology meaning, transformation code, configuration, and model version, then produce comparable inputs.

Every missing assumption becomes an improvement to the manifest. Reproducibility is a capability demonstrated by a drill, not a belief created by a version number.

A worked release sequence

Release 1.0 establishes the baseline source, ontology, split, and full review. Release 1.1 adds a batch of approved hard negatives under the same definitions. Release 1.2 corrects a documented boundary error and identifies affected experiments. Release 2.0 changes the ontology by splitting one broad class and requires migration plus new baselines.

Each release links to its parent and states whether downstream consumers must retrain, reevaluate, or take no action. The model registry records the exact release and transformation version. This turns data changes into a navigable history instead of a sequence of filenames.

Protect evaluation releases

Evaluation data should change less frequently than training data and require a higher approval bar. Keep it inaccessible to workflows that might introduce model-assisted leakage. Restrict who can view sensitive gold labels where needed.

When an evaluation correction is necessary, assess whether it changes historical conclusions. Rerun baselines that could be affected and communicate the new reference version clearly.

The complete path from working draft to model dependency

Start in Data Space with a dataset working draft. Curators add folders and assets, remove an ineligible batch, rename a source, or update tags. Those changes produce a Draft indicator; they do not rewrite the most recent published version. Publish version freezes the exact membership and group layout as DatasetVersion v3.

The project attaches v3 explicitly. Unitlab previews counts and deduplication, validates requirements such as video FPS, and creates independent project work. A later v4 may contain new assets, but v3 remains the project’s input until someone deliberately attaches the newer version. This prevents a curator’s ordinary work from changing an annotation queue midstream.

Inside the project, the Live ontology and workflow define label meaning and acceptance. Saving creates annotation history; stage actions route work. Invalid required values remain visible for correction instead of being discarded. Review approval moves the accepted item toward completion, while rejection returns it to the configured rework stage with reason and comment.

At release time, the owner defines title, visibility, split strategy, format, included files, and annotation snapshot. UUEF preserves the richest Unitlab structure, including multimodal groups and specialized geometry. Standard Bundle provides a general handoff, while family-specific formats are chosen only when they can represent the required annotations without silent loss. Source files and annotation packages can be downloaded separately where the workflow requires it.

The model configuration records both sides of lineage:

DatasetVersion v3
  → project copy
  → Live ontology version
  → workflow and review history
  → Release 2.1 / UUEF / split manifest
  → loader version + transformations
  → model run 8472

If a label correction arrives, the team changes the project state, reviews the fix, and creates Release 2.2. It does not alter Release 2.1. If source membership also changes, the team publishes a new DatasetVersion and records whether the project attached it. Dataset version and release version answer different questions and should not be collapsed into one number.

A reproducibility drill starts in a clean environment. Another engineer receives only the release reference, loader/transformation versions, ontology note, and model configuration. They download the exact package, verify counts and splits, and produce comparable model inputs. Any information they must ask for becomes a missing part of the contract.

Version history also supports controlled experimentation before release. A curator can inspect an immutable dataset snapshot, restore it into the current working draft, or duplicate it as a separate dataset. None of those actions rewrites the original version. A project can use View Source to inspect its exact attachment and Detach to remove only the project relationship while preserving source data and, by default, annotation history.

The release gate should reconcile counts across every boundary: source assets, DatasetVersion items, project work units, Data Groups, completed workflow items, accepted annotations, split membership, release rows, and downloaded records. A difference can be legitimate—an archived item or excluded split—but it needs a named reason. Silent count drift is a reproducibility failure even when every individual file is readable.

Record format limitations explicitly. Cuboids and deeply structured ontology data retain the richest representation in UUEF; a simpler family-specific export may omit concepts it cannot express natively. The correct response is to choose a suitable format or document a controlled transformation, not to assume every output preserves every Unitlab feature.

Reproduce one model result six months later

Imagine a model card that says, “trained on the March dataset.” Six months later, the dataset has new assets, corrected labels, a changed ontology, and different split rules. The phrase is operationally useless. Reproducibility begins when a model run references an immutable, described data release rather than a moving name.

For a new experiment, curate Assets into a dataset and create a version. Record selection filters, source windows, grouping logic, duplicate policy, split unit, and known exclusions. Attach the ontology and project instructions that defined label meaning. After annotation and review, create a Release from the approved dataset version and other required resources.

Store the Release identifier with the model configuration, code revision, seed, and evaluation result. The release notes should identify not only what was included but what changed from the prior release. A downstream reader should be able to tell whether a performance shift came from more data, corrected labels, new classes, source balance, or leakage repair.

Now rehearse the future question. Starting only from the model record, locate the release, inspect its component versions, recover the ontology meaning, and select the protected evaluation data. If the team must ask which folder was “current” that week, the chain is not complete. The exercise should work for a teammate who did not build the original dataset.

When a correction arrives, change the working dataset and create a new version. Build a new release instead of editing the historical one. Use cloning when exploring an alternative branch—such as a different class mapping or curation policy—so the experiment remains connected to its starting point without overwriting it.

Protect evaluation releases more strictly than exploratory training branches. Limit who can alter the source workflow, check duplicate leakage, and document every replacement. A model can improve because evaluation became easier; versioned releases make that possibility visible.

The six-month reproduction does not require every intermediate screen to be frozen forever. It requires the semantic inputs to be named: selected data, labels, ontology, instructions, grouping and split policies, review state, and release. When those are explicit, teams can compare model behavior as evidence instead of reconstructing it from memory.

Build a minimal provenance record for every run

At training time, store the Release identifier and description, dataset-version identifiers, ontology version, project or instruction reference, split policy, and known limitations beside the model configuration. For assisted annotation, include the model or prompt configuration that generated proposals when it materially affects label production.

At evaluation time, store the protected evaluation Release separately. Confirm that grouping units and duplicates do not cross the training boundary. If the evaluation release changes, create a new comparison line instead of replacing the old metric as though the benchmark were unchanged.

When results are reviewed, ask three questions. Did data membership change? Did label meaning or quality change? Did model configuration change? Versioned artifacts let the team answer each independently. Without that separation, a higher score may be attributed to modeling when it actually came from easier or leaked evaluation data.

Run the reproduction drill periodically, not only during an incident. Select an older model, resolve its release and ontology, inspect sample items, and recreate the evaluation cohort. Record any missing link as a process defect. Provenance becomes reliable through rehearsal in the same way disaster recovery does.

The objective is a chain a different team can follow. Personal notebooks may assist an experiment, but they cannot be the only place that explains which approved training data produced a deployed decision.

Treat changes as hypotheses

Every new release should state what changed and what effect is expected. Examples include adding night scenes to improve low-light recall, correcting a class mapping to reduce confusion, removing duplicates to produce a more honest evaluation, or publishing a new ontology version for a new task. This turns versioning into an experimental record rather than an archive of unexplained snapshots.

Compare the new model on the full protected evaluation release and on the cohort targeted by the change. A gain on the targeted cohort with a regression elsewhere is a decision, not an automatic success. Keep both results attached to the release and model record.

When several changes are combined—new data, new labels, and a new model architecture—acknowledge that attribution is limited. Controlled branches or cloned datasets can isolate important variables when the business decision requires it.

Record deprecation without erasure. An older release can be marked superseded or unsuitable for new training while remaining the correct reference for historical models. Deleting or rewriting it would destroy the evidence needed to understand those models.

Finally, include a human-readable summary. Machine identifiers provide precision; a short explanation tells future teams why the version exists. Both are necessary for durable reproducibility.

Assign an owner and retention expectation to important releases. The owner answers consumer questions and approves supersession; retention protects the evidence required for deployed or externally reported models. Storage policy should distinguish removable working artifacts from release records that remain part of model provenance.

If source data must be removed for legal or contractual reasons, record the effect on reproducibility and affected models rather than silently breaking the chain. Versioning cannot eliminate every external constraint, but it can make the consequence visible.

A release-to-model operating contract

Before training, the data owner supplies one Release identifier, its component dataset versions, ontology version, grouping and split policy, review state, and change notes. The model owner records that identifier beside code, configuration, seed, environment, and output metrics. No run should depend only on a path called latest.

During training, record any transformation that changes membership or meaning: sampling, filtering, augmentation exclusions, class remapping, frame extraction, or missing-modality handling. If the transformation is expected to be reused, promote it into a governed dataset or release process rather than leaving it in an untracked notebook.

During evaluation, resolve a separate protected Release. Verify duplicate and group boundaries against training. Keep the evaluation identifier with every metric. If evaluation data changes, report a new benchmark line and preserve the old one.

After results, compare the run with its predecessor by data membership, label or ontology meaning, and model configuration. If multiple dimensions changed, state that attribution is mixed. Create controlled clones or branches when isolating a data change matters to the decision.

At promotion, record the deployed model's training and evaluation releases. If the model is rolled back, the data references roll back with it. If a data defect is found, identify every model that consumed the affected release, correct forward, and publish a new release.

Run a quarterly reproduction drill: choose an older model, retrieve both releases, inspect samples and schema, and recalculate a small evaluation. Every missing link becomes an owned process issue. Reproducibility is not a promise stored in metadata; it is a workflow proven by another person.

Try the workflow in Unitlab

Make the release record useful to someone who was not in the meeting. Include the source dataset version, ontology version, project or Batch Queue scope, exclusions, split policy, annotation state, export format, and approval rationale. A version number identifies the artifact; this context explains why it is fit for a particular model run.

Then perform a retrieval drill. Starting only from a model experiment record, locate the exact Unitlab release, inspect its immutable inputs, and recreate the export without selecting “latest” anywhere. If that path depends on memory or a private message, the lineage is incomplete.

Corrections belong in a new working branch and a new release. Preserve the prior release even when the error is embarrassing: historical model behavior is reproducible only when the data it consumed remains addressable. Compare the two releases explicitly and document whether metrics changed because of corrected labels, changed population, or both.

That comparison becomes part of the model card or experiment record.

1. Choose a dataset state that has completed its agreed review and schema checks.
2. Record selection, ontology version, workflow, source period, splits, transformations, counts, and exclusions.
3. Create a Unitlab release and inspect its version selector, sample data, metadata, and ground truth.
4. Download or clone the exact release into a clean environment and reproduce the consumer load.
5. Link the release ID to the model run, evaluation, and change note; never overwrite the historical reference.

Audit reproducibility, count reconciliation, schema compatibility, split leakage, undocumented transformation, and consumer workarounds. A version is complete only when someone else can use it without reconstructing hidden steps.

The decision to make

Take the next model run off a mutable folder. Create a Unitlab release, record its contract, and make the release ID part of the experiment metadata.