treeverse / treeverse/dvc

`dvc queue start` checkout more files than required

Open
#10,293 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A: experiments p2-medium performance
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Bug Report / Feature Request

I have a project with a data directory (150 GB) containing 11 files. I have added the entire directory using dvc add data.
In my workflow each experiment I want to conduct only depends on a single file in the data directory.
Running an experiment using the dvc queue will dvc checkout the entire data directory.
It would be much faster if the command only dvc checkout data files, which are actually required by the workflow, as defined in the dvc.yaml .

Expected

The data directory in .dvc/tmp/exp/... would only contain files, specified as explicit dependencies in the dvc.yaml workflow file.

Environment information

Output of dvc doctor:

DVC version: 3.43.1 (pip)
-------------------------
Platform: Python 3.11.7 on Linux-6.5.0-15-generic-x86_64-with-glibc2.35
Subprojects:
        dvc_data = 3.9.0
        dvc_objects = 3.0.6
        dvc_render = 1.0.1
        dvc_task = 0.3.0
        scmrepo = 2.0.2
Supports:
        http (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.9.1, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2024.2.0, boto3 = 1.34.34)
Config:
        Global: /tikhome/fzills/.config/dvc
        System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: nfs on 129.69.120.13:/share/work_icp/fzills
Caches: local
Remotes: None
Workspace directory: nfs on 129.69.120.13:/share/work_icp/fzills
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/240bb452ebd33bc5c31f30d78040c7d2

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how dvc queue invokes dvc checkout for experiment workspaces under .dvc/tmp/exp/..., then inspect how dependencies from dvc.yaml are resolved. Reproduce with a directory dependency containing multiple files and a workflow that uses one explicit file. Done means the experiment workspace contains only the required data files rather than the entire directory.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.