treeverse / treeverse/dvc

repro: --allow-missing skips changed dependencies inside directory outputs

Open
#11,099 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Bug Report

Description

When one stage outputs a directory and another depends on a file inside it, dvc repro --allow-missing --dry can report everything up to date even though that file changed upstream. This happens when the directory is missing from the workspace, even with its current .dir manifest available in the local cache.

This makes the command unreliable for CI freshness checks without checking out the data.

Reproduce

With DVC 3.67.1 installed, run in Bash:

set -eu
repro_dir=$(mktemp -d)
cd "$repro_dir"
git init -q
dvc init -q
dvc config --local cache.dir .dvc/cache

echo old > source.txt
dvc stage add -n produce -d source.txt -o data \
  'mkdir -p data && cp source.txt data/file.txt'
dvc stage add -n consume -d data/file.txt -o result.txt \
  'cp data/file.txt result.txt'
dvc repro

echo new > source.txt
dvc repro produce
mv data data.saved
dvc repro --allow-missing --dry

Actual output from the last command:

Stage 'produce' didn't change, skipping
Stage 'consume' didn't change, skipping
Data and pipelines are up to date.

Restoring the same directory makes DVC correctly detect that consume needs reproduction:

mv data.saved data
dvc repro --allow-missing --dry
Expected

The dry run should identify consume as needing reproduction in both cases. Its recorded dependency hash differs from the file's hash in the updated directory manifest; the file contents should not need to be checked out to detect this.

Note: Related to #9530, fixed by #9533 for dependencies whose paths exactly match upstream outputs. This reproduction covers a remaining case: a dependency on a file inside an upstream directory output. It still reproduces on DVC 3.67.1.

Environment information
Output of dvc doctor (personal paths redacted)
DVC version: 3.67.1 (pip)
-------------------------
Platform: Python 3.13.12 on Linux-6.8.0-138-generic-x86_64-with-glibc2.39
Subprojects:
    dvc_data = 3.18.3
    dvc_objects = 5.2.0
    dvc_render = 1.0.2
    dvc_task = 0.40.2
    scmrepo = 3.6.2
Supports:
    http (aiohttp = 3.14.3, aiohttp-retry = 2.9.1),
    https (aiohttp = 3.14.3, aiohttp-retry = 2.9.1)
Config:
    Global: [redacted]/.config/dvc
    System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: zfs on fastnvme/workspace-tmp
Caches: local
Remotes: None
Workspace directory: zfs on fastnvme/workspace-tmp
Repo: dvc, git
Repo.site_cache_dir: [redacted]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the Bash reproduction with DVC 3.67.1 and inspect the dependency-hash handling used by dvc repro --allow-missing --dry for files inside directory outputs. Done means the final dry run identifies consume as needing reproduction when data is absent but its updated .dir manifest is available in the cache.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, data
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.