iterative / iterative/example-repos-dev

Tutorial proposal: compare segmentation augmentation policies with DVC Experiments

Open
#285 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
21
Forks
13
PR merge metrics
No merged PRs in 30d

Description

Would you be open to a small standalone tutorial showing how DVC Experiments can compare two image-augmentation policies for the same segmentation task, including the exact derived training data, representative image/mask previews, and the resulting Dice score?

## User problem

Changing an augmentation policy changes the effective training dataset. A metric alone does not show which labeled samples the model actually saw, and an image preview alone does not preserve the dataset and policy that produced the result.

The current [`example-get-started-experiments`](https://github.com/iterative/example-repos-dev/tree/69d9e194c81914c92d8e5b3945998416f338b7cd/example-get-started-experiments) already demonstrates the important DVC pieces for the swimming-pool segmentation task: a staged pipeline, parameters, cached outputs, experiment queues, Dice evaluation, and DVCLive image plots. Issue [#188](https://github.com/iterative/example-repos-dev/issues/188) also established that the test set must remain fixed for metrics and prediction images to be comparable across experiments.

The proposed tutorial would keep that test set and evaluation unchanged. It would vary only a versioned training-data preparation stage.

## Proposed workflow

1. Add an optional `augment` stage after `data_split`. It reads each training image/mask pair and writes a materialized derived dataset.
2. Select a bounded `light` or `strong` policy through DVC parameters.
3. Use AlbumentationsX to apply each sampled geometric transform to the image and mask together. A stable seed derived from the sample ID and variant index is passed as `invocation_seed`, so rerunning the stage produces the same files independently of processing order.
4. Validate that every output image/mask pair has matching spatial dimensions and that each mask still contains only the declared segmentation labels.
5. Save the resolved policy from `policy.to_dict()` next to the derived dataset and log a small fixed set of original/augmented image-and-mask previews with DVCLive.
6. Train and evaluate the same model for both policies. DVC then connects the selected parameters, stage dependencies, cached dataset output, previews, and Dice metric for each experiment.

A compact stage boundary could look like this:

```yaml
stages:
augment:
cmd: python src/augment.py
deps:
- data/train_data
- src/augment.py
params:
- base.random_seed
- augmentation
outs:
- data/train_augmented
- results/augment
```

The AX call inside that stage remains explicit:

```python
augmented = policy(
image=image,
mask=mask,
invocation_seed=sample_seed,
)
```

The user can then answer two related questions from one experiment comparison: what did this policy do to the labeled training data, and how did that dataset change affect the fixed evaluation?

## Scope and placement

This would be a tutorial/example contribution. It requires no DVC or DVCLive API change.

The augmentation stage is independent of the training framework, so the proposal does not depend on the current fastai implementation and can coexist with the migration discussed in [#255](https://github.com/iterative/example-repos-dev/issues/255).

Would maintainers prefer this as:

- an optional advanced variant of `example-get-started-experiments`;
- a separate generated example repository; or
- a focused dvc.org tutorial that leaves the foundational example unchanged?

If the workflow fits the project, I can prepare the tutorial after your guidance on the surface and training-framework target.

## Optional dependency boundary

The example would install AlbumentationsX only in its own environment. The current public package is AGPL-3.0-only and requires Python 3.10 or newer. Users install the PyTorch build appropriate for CPU, CUDA, or MPS before AlbumentationsX; PyTorch is intentionally not selected through the AlbumentationsX package metadata.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing example-get-started-experiments and the fixed-test-set decision in issue #188, then confirm with maintainers whether this belongs in that example, a separate repository, or dvc.org. Completion would be a standalone tutorial with the proposed augmentation stage, reproducible derived data and previews, policy parameters, and comparable Dice results for both policies.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, documentation, machine-learning
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.