treeverse / treeverse/dvc

dvc exp run --temp: dvc-tracked dependencies are not checked out

Open
#10,056 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Bug Report

Issue name

dvc exp run --temp: dvc-tracked dependencies are not checked out

Description

dvc exp run --temp will not dvc checkout the dependency file.txt from file.txt.dvc if file.txt has been modified.

Reproduce
  1. Setup a git repo with the following minimal structure:
├── code
│   └── test.py
├── data
│   └── file.txt
├── dvc.yaml
└── README.md

code/test/py just prints the contents the file.txt, and file.txt contains the single line "foo". file.txt is gitignored, all other files are git tracked. Content of dvc.yaml is:

❯ cat dvc.yaml
stages:
  generate_data:
    cmd: python3 code/test.py
    deps:
      - data/file.txt
  1. dvc init && dvc add data/file.txt && git add data/file.txt.dvc data/.gitignore
  2. dvc exp run --dry
  3. dvc exp run --temp -> prints "foo", as expected.
  4. rm data/file.txt && dvc exp run --temp -> checks out file.txt and prints "foo" as expected.
'data/file.txt.dvc' didn't change, skipping                                                                                   
Running stage 'generate_data':                                                                                                
> python3 code/test.py
foo
  1. touch data/file.txt && dvc exp run --temp -> fails to dvc check out file.txt and runs with empty file.txt instead
WARNING: 'data/file.txt' is empty.                                                                                            
Running stage 'generate_data':
> python3 code/test.py

Expected

dvc exp run --temp should not copy file.txt to the temporary folder, since it is not git tracked, and dvc checkout file.txt from the local cache, just as it does when file.txt is not present at all.

Environment information

dvc==3.27.0

Output of dvc doctor:

❯ dvc doctor
DVC version: 3.27.0 (pip)
-------------------------
Platform: Python 3.8.12 on Linux-5.15.0-87-generic-x86_64-with-glibc2.17
Subprojects:
        dvc_data = 2.18.2
        dvc_objects = 1.0.1
        dvc_render = 0.5.3
        dvc_task = 0.3.0
        scmrepo = 1.4.0
Supports:
        http (aiohttp = 3.8.1, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.1, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.6.0, boto3 = 1.26.76),
        ssh (sshfs = 2023.4.1)
Config:
        Global: /home/ralf/.config/dvc
        System: /etc/xdg/xdg-ubuntu/dvc
Cache types: <https://error.dvc.org/no-dvc-cache>
Caches: local
Remotes: None
Workspace directory: ext4 on /dev/nvme0n1p5
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/cae49d9183304046b1da8acdeb50f42f

Additional Information (if any):

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the issue with the listed dvc exp run --temp commands, especially after touching data/file.txt, then trace the temporary experiment setup and checkout handling. Done means a modified DVC-tracked dependency is restored from the local cache before the stage runs, matching the behavior when the file is absent.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.