.dvcignore isn't followed when dvc is in `--subdir` mode
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.9k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
Bug Report
repro: Doesn't follow .dvcignore when DVC is initialized in --subdir mode and the dependency is outside the DVC subdir repo
Description
In a monorepo, I have shared Python code at the Git repository root and multiple project directories that each initialize DVC independently with dvc init --subdir.
Example structure:
.
|-- .dvcignore
|-- main_lib/
|-- pyproject.toml
`-- sudirs_monorepo/
|-- proj1/
| |-- .dvcignore
| |-- .hashes/
| | `-- dvc.yaml
| |-- proj1/
| `-- pyproject.toml
`-- proj2/
|-- .dvcignore
|-- .hashes/
| `-- dvc.yaml
|-- proj2/
`-- pyproject.toml
All .dvcignore files contain:
*.bz2
*.pyc
The stage in sudirs_monorepo/proj1/.hashes/dvc.yaml depends on the root-level main_lib/ directory:
stages:
check_main_code:
desc: "Detects changes in any python file in main_lib."
wdir: ../
cmd: 'git ls-files -sc ''../../main_lib/*.py'' > ".hashes/main_lib_code.hash"'
deps:
- ../../main_lib/
outs:
- .hashes/main_lib_code.hash:
cache: true
persist: false
If DVC is initialized at the Git repository root, dvc repro sudirs_monorepo/proj1/.hashes/dvc.yaml correctly ignores generated *.pyc files and reports that the stage did not change.
If DVC is initialized inside each project with dvc init --subdir, running dvc --cd sudirs_monorepo/proj1 repro .hashes/dvc.yaml treats the dependency as changed after Python bytecode files are generated under main_lib/**/__pycache__/, even though *.pyc is ignored.
Reproduce
Start from a clean checkout where the baseline dvc.lock was created before any __pycache__ files existed.
Control case, DVC initialized at the Git repository root:
$ dvc init
$ dvc repro sudirs_monorepo/proj1/.hashes/dvc.yaml
$ python -m compileall --invalidation-mode checked-hash main_lib/
$ dvc repro sudirs_monorepo/proj1/.hashes/dvc.yaml
Observed:
Stage 'sudirs_monorepo/proj1/.hashes/dvc.yaml:check_main_code' didn't change, skipping
Data and pipelines are up to date.
Bug case, DVC initialized from the project subdir:
$ dvc --cd sudirs_monorepo/proj1 init --subdir
$ dvc --cd sudirs_monorepo/proj1 repro .hashes/dvc.yaml
$ python -m compileall --invalidation-mode checked-hash main_lib/
$ dvc --cd sudirs_monorepo/proj1 repro .hashes/dvc.yaml
Observed:
Running stage '.hashes/dvc.yaml:check_main_code':
> git ls-files -sc '../../main_lib/*.py' > ".hashes/main_lib_code.hash"
Verbose output shows that only the dependency hash changed:
DEBUG: built tree 'object a6b66a993427ece6fde5ee6109ee5241.dir'
DEBUG: Dependency '...\main_lib' of stage: '.hashes/dvc.yaml:check_main_code' changed because it is 'modified'.
DEBUG: stage: '.hashes/dvc.yaml:check_main_code' changed.
The baseline lock, created before bytecode files existed, only contains the Python source files:
deps:
- path: ../../main_lib/
hash: md5
md5: f34a49f833b47ec363053c44e100aba8.dir
size: 2676
nfiles: 16
After compileall, the only new files under main_lib/ are *.pyc files inside __pycache__ directories.
Expected
The subdir-initialized DVC repo should behave like the root-initialized DVC repo.
When the only changes under ../../main_lib/ are ignored *.pyc files, this command should skip the stage:
$ dvc --cd sudirs_monorepo/proj1 repro .hashes/dvc.yaml
Expected output:
Stage '.hashes/dvc.yaml:check_main_code' didn't change, skipping
Data and pipelines are up to date.
DVC should either respect the applicable .dvcignore rules when hashing dependencies outside the --subdir DVC repo root, or provide a supported way for a subdir DVC repo to ignore files in such external dependencies.
Environment information
Output of dvc doctor:
$ dvc doctor
DVC version: 3.67.1 (pip)
-------------------------
Platform: Python 3.13.13 on Windows-11-10.0.22631-SP0
Subprojects:
dvc_data = 3.18.3
dvc_objects = 5.2.0
dvc_render = 1.0.2
dvc_task = 0.40.2
scmrepo = 3.6.2
Supports:
http (aiohttp = 3.13.5, aiohttp-retry = 2.9.1),
https (aiohttp = 3.13.5, aiohttp-retry = 2.9.1)
Config:
Global: C:\Users\<user>\AppData\Local\iterative\dvc
System: C:\ProgramData\iterative\dvc
Cache types: hardlink
Cache directory: NTFS on C:\
Caches: local
Remotes: None
Workspace directory: NTFS on C:\
Repo: dvc, git
Repo.site_cache_dir: C:\ProgramData\iterative\dvc\Cache\repo\889e6d9e2b515e0190f5a307725a8fde
Additional Information (if any):
The difference is visible before the stage command itself matters: in the --subdir case, DVC computes a different dependency tree for ../../main_lib/ after python -m compileall --invalidation-mode checked-hash main_lib/, while the root-initialized DVC repo keeps the original dependency tree hash and skips the stage. This issue is for all operating systems and not limited to windows where I have shared the reproducing steps.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the difference between root-initialized DVC and dvc init --subdir using the provided dvc repro commands, then inspect how dependency trees are hashed for ../../main_lib/. Compare the applicable .dvcignore handling after generating *.pyc files under __pycache__. Done means the subdir command skips the stage when only ignored bytecode files change, matching the root-initialized behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100