pyronear / pyronear/temporal-model
eval: track data/01_raw/datasets via a frozen DVC import instead of a manual rsync from train
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1
- Forks
- 0
- Avg merge
- 12h 43m
- Merged PRs (30d)
- 7
Description
eval/data/01_raw/datasets/{train,val} (~5.9 GB, ~97k files) is a dep of evaluate@train/@val but is not tracked by any .dvc file and not produced by any eval stage — it is refreshed by hand-rsyncing train/data/01_raw/datasets/ (now documented in docs/runbooks/retrain-on-new-dataset.md step 3, but still manual).
Failure modes:
- Skip or mistime the rsync and eval silently scores the new model against the previous release's truncated data; the run succeeds and the lock records a self-consistent hash, so nothing distinguishes it from a correct run.
dvc pushnever uploads these files, so a fresh clone / CI cannot reproduce the eval pipeline without cloning train and re-runningtruncate.
Proposed fix: mirror the pattern already used for model.zip one bullet down — a frozen local dvc import of train's truncate outputs plus an update-datasets make target, making the dep pushable, pullable, and reviewable. Noticed in the PR #65 review; kept out of that PR to avoid scope creep.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Read docs/runbooks/retrain-on-new-dataset.md step 3 and inspect the existing model.zip import pattern in the repository. Trace how eval/data/01_raw/datasets/{train,val} is consumed by evaluate@train/@val, then define the frozen import and update-datasets target. Done means the dataset is represented by repository metadata and can be pushed, pulled, and reproduced without manual rsyncing or cloning train.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- build-system, data-engineering
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 65/100