pyronear / pyronear/temporal-model

eval: track data/01_raw/datasets via a frozen DVC import instead of a manual rsync from train

Open
#67 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
1
Forks
0
Avg merge
12h 43m
Merged PRs (30d)
7

Description

eval/data/01_raw/datasets/{train,val} (~5.9 GB, ~97k files) is a dep of evaluate@train/@val but is not tracked by any .dvc file and not produced by any eval stage — it is refreshed by hand-rsyncing train/data/01_raw/datasets/ (now documented in docs/runbooks/retrain-on-new-dataset.md step 3, but still manual).

Failure modes:

  1. Skip or mistime the rsync and eval silently scores the new model against the previous release's truncated data; the run succeeds and the lock records a self-consistent hash, so nothing distinguishes it from a correct run.
  2. dvc push never uploads these files, so a fresh clone / CI cannot reproduce the eval pipeline without cloning train and re-running truncate.

Proposed fix: mirror the pattern already used for model.zip one bullet down — a frozen local dvc import of train's truncate outputs plus an update-datasets make target, making the dep pushable, pullable, and reviewable. Noticed in the PR #65 review; kept out of that PR to avoid scope creep.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read docs/runbooks/retrain-on-new-dataset.md step 3 and inspect the existing model.zip import pattern in the repository. Trace how eval/data/01_raw/datasets/{train,val} is consumed by evaluate@train/@val, then define the frozen import and update-datasets target. Done means the dataset is represented by repository metadata and can be pushed, pulled, and reproduced without manual rsyncing or cloning train.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
build-system, data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.