treeverse / treeverse/dvc

`dvc repro -R <dir_1>` can run each `dir_1/**/dvc.yaml` from CWD

Open
#10,503 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

A: pipelines feature request
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

For a repository with many stages, the dvc.yaml file can become very large. It can be useful in these cases to split it out into smaller versions for readibility, e.g. into separate dir_1/dvc.yaml, dir_2/dvc.yaml, etc.

For the same stages to run, the stage yamls all need to be updated to take into account their new location, e.g. with wdir: "..", which can be cumbersome when many files are moved.

It would be nice to be able to customise the dvc repro -R <dir_name> command, e.g. with dvc repro -R -c=. <dir_name>, to run from a customisable location without specifying relative folder locations within the individual dvc.yaml stages.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the dvc repro -R <dir_name> command and how it resolves dvc.yaml files and working directories. Define how a customizable location such as -c=. should affect recursive reproduction without requiring relative wdir changes, then verify behavior with a repository containing multiple nested dvc.yaml files.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.