treeverse / treeverse/dvc

`dvc diff <rev>` reports all-deleted/0-added for outputs after splitting a single-dir `.dvc` into per-subfolder `.dvc` files (3.66.1)

Open
#11,050 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Description

After refactoring a single dir-output .dvc (e.g. review.dvc tracking
review/) into multiple sub-folder .dvc files (review/<name>.dvc,
each with path: <name> relative to its parent), dvc diff HEAD
(i.e. HEAD vs workspace) reports every file under the refactored
subtree as deleted, with 0 added and 0 modified, even though:

  • dvc status reports the workspace fully in sync.
  • The files all exist on disk at the expected paths.
  • The dir-hash .dir cache files referenced by each new .dvc are
    present and enumerate correctly.
  • HEAD's .dvc files are byte-identical to workspace's .dvc files
    (git status clean).
  • dvc diff HEAD HEAD (rev-to-rev) is correctly empty.
  • dvc diff HEAD~1 HEAD (other rev-to-rev pairings) work correctly.
  • dvc diff HEAD --targets <unaffected-subtree> works correctly.

The zero-added/zero-modified counts suggest the workspace-side
enumeration is returning an empty file list for the refactored
outputs, while the HEAD-side enumeration correctly expands the dir
caches; the diff then attributes all HEAD-side files to "deleted."

Reproduction

Pre-refactor state:

project/
  review.dvc            # outs: [{path: review, md5: <dir-hash>}]
  review/
    after_cma_phase1/
    clinician_files/
    ...

Refactor (each commit does one):

# Remove the umbrella tracker, leaving files in place
dvc remove review.dvc
git rm review.dvc

# Track each subfolder individually
dvc add review/after_cma_phase1
dvc add review/clinician_files
dvc add review/edf_files
# ...
git add review/*.dvc
git commit

Each new review/<name>.dvc looks like:

```yaml
outs:

  • md5: 69c24de62577c8e011842eaaa013ffc6.dir
    size: 1195692789
    nfiles: 140
    hash: md5
    path: after_cma_phase1
    ```

Then:

```bash
$ dvc status
Data and pipelines are up to date.

$ dvc diff HEAD HEAD

(empty, as expected)

$ dvc diff HEAD HEAD~1

(correct diff for the actual rev-to-rev change)

$ dvc diff HEAD --json | jq '{added, deleted, modified} | map_values(length)'
{ "added": 0, "deleted": 11957, "modified": 0 }
```

All 11,957 "deleted" paths still exist on disk and are correctly
listed inside their respective .dir cache files.

Things tried that did not help
  • dvc commit -f (no-op since dvc status is clean).
  • Running dvc diff HEAD from the subdirectory containing the
    refactored .dvc files.
  • Verifying every .dir cache file for each new sub-folder .dvc
    is present and well-formed.
Workarounds
  • dvc status for sync checks (authoritative; not affected).
  • Use commit-to-commit dvc diff <rev1> <rev2> (not affected).
  • Scope HEAD-vs-workspace diffs to unaffected targets:
    dvc diff HEAD --targets <other-target> (not affected).
Expected behavior

dvc diff HEAD with no workspace changes should report an empty
diff, matching dvc status.

Environment

```
DVC version: 3.66.1 (brew)
CPython 3.14.2
Linux 6.18.28-1-lts (x86_64)
glibc 2.43
```

Repo layout: monorepo, DVC project lives in a subdirectory of the
git repo (git toplevel and DVC root are not the same path). Remote:
local FS path remote.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the implementation and tests for dvc diff HEAD, comparing workspace-side enumeration with revision-side enumeration for .dvc outputs and their .dir caches. Reproduce the refactor using the commands in the issue and verify that an unchanged workspace reports zero added, deleted, and modified files while revision-to-revision diffs remain correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli, data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.