repro: Rebuilds same tree unnecessarily
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.9k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
Given a directory tracked with dvc add/dvc import and a dvc.yaml with stages that have that directory as dependeny:
$ cat data.dvc
outs:
- md5: 6f68a8a747e41c152e7cc5fc62437727.dir
size: 2890
nfiles: 1000
path: data
$ cat stages:
foo:
cmd: echo foo
deps:
- data
bar:
cmd: echo bar
deps:
- data
During a dvc repro execution, the same tree for the .dir is being built (_build_tree) multiple times during:
changed_outsfordata.dvc
Unless I am missing something, this is the only place where we should really call_build_treeand cache the result.- (for each stage)
changed_deps - (for each stage)
save_depsas part of_run_stage. - (for each stage)
save_depsas part ofsave
I don't really know why we need to callsave_depstwice insidestage.run.
So, in total there are 3 unnecessary (IMO) calls to _build_tree for each stage.
For 100k dummy files, each of these _build_tree calls takes around 10s.
It feels like a significant overhead, especially considering that it grows with the number of files and the number of stages having them as deps.
Don't know if this is something to be addressed in https://github.com/iterative/dvc-data or in DVC as part of pipeline management
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with dvc/stage/init.py, especially changed_deps, _run_stage, and save, then inspect dvc-data's _build_tree in src/dvc_data/hashfile/build.py. Reproduce the directory dependency case with dvc repro and measure repeated tree builds. Done means avoiding the unnecessary per-stage rebuilds while preserving dependency checking and stage execution behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, performance, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100