treeverse / treeverse/dvc

.dvcignore isn't followed when dvc is in `--subdir` mode

Open
#11,044 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Bug Report

repro: Doesn't follow .dvcignore when DVC is initialized in --subdir mode and the dependency is outside the DVC subdir repo

Description

In a monorepo, I have shared Python code at the Git repository root and multiple project directories that each initialize DVC independently with dvc init --subdir.

Example structure:

.
|-- .dvcignore
|-- main_lib/
|-- pyproject.toml
`-- sudirs_monorepo/
    |-- proj1/
    |   |-- .dvcignore
    |   |-- .hashes/
    |   |   `-- dvc.yaml
    |   |-- proj1/
    |   `-- pyproject.toml
    `-- proj2/
        |-- .dvcignore
        |-- .hashes/
        |   `-- dvc.yaml
        |-- proj2/
        `-- pyproject.toml

All .dvcignore files contain:

*.bz2
*.pyc

The stage in sudirs_monorepo/proj1/.hashes/dvc.yaml depends on the root-level main_lib/ directory:

stages:
  check_main_code:
    desc: "Detects changes in any python file in main_lib."
    wdir: ../
    cmd: 'git ls-files -sc ''../../main_lib/*.py'' > ".hashes/main_lib_code.hash"'
    deps:
      - ../../main_lib/
    outs:
      - .hashes/main_lib_code.hash:
          cache: true
          persist: false

If DVC is initialized at the Git repository root, dvc repro sudirs_monorepo/proj1/.hashes/dvc.yaml correctly ignores generated *.pyc files and reports that the stage did not change.

If DVC is initialized inside each project with dvc init --subdir, running dvc --cd sudirs_monorepo/proj1 repro .hashes/dvc.yaml treats the dependency as changed after Python bytecode files are generated under main_lib/**/__pycache__/, even though *.pyc is ignored.

Reproduce

Start from a clean checkout where the baseline dvc.lock was created before any __pycache__ files existed.

Control case, DVC initialized at the Git repository root:

$ dvc init
$ dvc repro sudirs_monorepo/proj1/.hashes/dvc.yaml
$ python -m compileall --invalidation-mode checked-hash main_lib/
$ dvc repro sudirs_monorepo/proj1/.hashes/dvc.yaml

Observed:

Stage 'sudirs_monorepo/proj1/.hashes/dvc.yaml:check_main_code' didn't change, skipping
Data and pipelines are up to date.

Bug case, DVC initialized from the project subdir:

$ dvc --cd sudirs_monorepo/proj1 init --subdir
$ dvc --cd sudirs_monorepo/proj1 repro .hashes/dvc.yaml
$ python -m compileall --invalidation-mode checked-hash main_lib/
$ dvc --cd sudirs_monorepo/proj1 repro .hashes/dvc.yaml

Observed:

Running stage '.hashes/dvc.yaml:check_main_code':
> git ls-files -sc '../../main_lib/*.py' > ".hashes/main_lib_code.hash"

Verbose output shows that only the dependency hash changed:

DEBUG: built tree 'object a6b66a993427ece6fde5ee6109ee5241.dir'
DEBUG: Dependency '...\main_lib' of stage: '.hashes/dvc.yaml:check_main_code' changed because it is 'modified'.
DEBUG: stage: '.hashes/dvc.yaml:check_main_code' changed.

The baseline lock, created before bytecode files existed, only contains the Python source files:

deps:
- path: ../../main_lib/
  hash: md5
  md5: f34a49f833b47ec363053c44e100aba8.dir
  size: 2676
  nfiles: 16

After compileall, the only new files under main_lib/ are *.pyc files inside __pycache__ directories.

Expected

The subdir-initialized DVC repo should behave like the root-initialized DVC repo.

When the only changes under ../../main_lib/ are ignored *.pyc files, this command should skip the stage:

$ dvc --cd sudirs_monorepo/proj1 repro .hashes/dvc.yaml

Expected output:

Stage '.hashes/dvc.yaml:check_main_code' didn't change, skipping
Data and pipelines are up to date.

DVC should either respect the applicable .dvcignore rules when hashing dependencies outside the --subdir DVC repo root, or provide a supported way for a subdir DVC repo to ignore files in such external dependencies.

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 3.67.1 (pip)
-------------------------
Platform: Python 3.13.13 on Windows-11-10.0.22631-SP0
Subprojects:
	dvc_data = 3.18.3
	dvc_objects = 5.2.0
	dvc_render = 1.0.2
	dvc_task = 0.40.2
	scmrepo = 3.6.2
Supports:
	http (aiohttp = 3.13.5, aiohttp-retry = 2.9.1),
	https (aiohttp = 3.13.5, aiohttp-retry = 2.9.1)
Config:
	Global: C:\Users\<user>\AppData\Local\iterative\dvc
	System: C:\ProgramData\iterative\dvc
Cache types: hardlink
Cache directory: NTFS on C:\
Caches: local
Remotes: None
Workspace directory: NTFS on C:\
Repo: dvc, git
Repo.site_cache_dir: C:\ProgramData\iterative\dvc\Cache\repo\889e6d9e2b515e0190f5a307725a8fde

Additional Information (if any):

The difference is visible before the stage command itself matters: in the --subdir case, DVC computes a different dependency tree for ../../main_lib/ after python -m compileall --invalidation-mode checked-hash main_lib/, while the root-initialized DVC repo keeps the original dependency tree hash and skips the stage. This issue is for all operating systems and not limited to windows where I have shared the reproducing steps.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the difference between root-initialized DVC and dvc init --subdir using the provided dvc repro commands, then inspect how dependency trees are hashed for ../../main_lib/. Compare the applicable .dvcignore handling after generating *.pyc files under __pycache__. Done means the subdir command skips the stage when only ignored bytecode files change, matching the root-initialized behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, tooling
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.