treeverse / treeverse/dvc

repro: Rebuilds same tree unnecessarily

Open
#9,085 6 comments 8 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A: pipelines p1-important performance
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Given a directory tracked with dvc add/dvc import and a dvc.yaml with stages that have that directory as dependeny:

$ cat data.dvc
outs:
- md5: 6f68a8a747e41c152e7cc5fc62437727.dir
  size: 2890
  nfiles: 1000
  path: data
$ cat stages:
  foo:
    cmd: echo foo
    deps:
    - data
  bar:
    cmd: echo bar
    deps:
    - data

During a dvc repro execution, the same tree for the .dir is being built (_build_tree) multiple times during:

  • changed_outs for data.dvc
    Unless I am missing something, this is the only place where we should really call _build_tree and cache the result.
  • (for each stage) changed_deps
  • (for each stage) save_deps as part of _run_stage.
  • (for each stage) save_deps as part of save
    I don't really know why we need to call save_deps twice inside stage.run.

So, in total there are 3 unnecessary (IMO) calls to _build_tree for each stage.

For 100k dummy files, each of these _build_tree calls takes around 10s.

It feels like a significant overhead, especially considering that it grows with the number of files and the number of stages having them as deps.

Don't know if this is something to be addressed in https://github.com/iterative/dvc-data or in DVC as part of pipeline management

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with dvc/stage/init.py, especially changed_deps, _run_stage, and save, then inspect dvc-data's _build_tree in src/dvc_data/hashfile/build.py. Reproduce the directory dependency case with dvc repro and measure repeated tree builds. Done means avoiding the unnecessary per-stage rebuilds while preserving dependency checking and stage execution behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, performance, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.