treeverse / treeverse/dvc

Add `--num` option to `dvc fetch`

Open
#11,033 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Problem
When merging two branches in a DVC repository it is possible to encounter missing cache data from previous commits. This results in errors such as:
ERROR: unexpected error - [Errno 2] No such file or directory: '~/my_repo/.dvc/cache/files/md5/xx/xxx.dir'

To resolve this, the current workaround is to run:
dvc fetch --all-commits
on both branches to ensure all required data is available locally. However, for long-lived branches like main, this can result in downloading a very large amount of data, much of which is not needed.

Proposed solution
Introduce a --num <N> option to dvc fetch, similar to the existing option in dvc gc, which limits operations to the most recent N commits.

Example:
dvc fetch --num 10
This would fetch cache data only for the latest 10 commits in the current branch, reducing unnecessary downloads.

I would be interested in contributing an implementation if this feature is considered useful.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the CLI implementation for dvc fetch and the existing --num option in dvc gc. Trace how --all-commits selects commits and how fetch downloads their cache data. Done means dvc fetch --num N limits fetching to the latest N commits in the current branch, with coverage for the new option.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.