checkout and fetch: .dvc stages are not resolved correctly when using --with-deps
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.9k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
Bug Report
Description
- When I
dvc fetcha subdirectory of animported asset (tracked by.dvcfile), I see a "N files fetched" message. However, doingdvc checkoutof the same subdir has no effect (nothing checked out in workspace).- As might be expected, I see the same behavior if I do
dvc pullinstead. - If the asset is an
added asset instead of animported one, all files in the asset are fetched
- As might be expected, I see the same behavior if I do
- Related, when I
dvc fetch --with-depsa stage that has adepwhich is a subdirectory of an import, it doesn't fetch the data for that dependency.- I traced down the code and it looks like the dependency gets lost in the call to
repo.used_objs() - This happens both in versions 2.58.2 and 2.53.0.
- If I drop down to 2.52.0 then fetch goes back to the old behavior of pulling too much data 😂 (see #6124 and #9246)
- I traced down the code and it looks like the dependency gets lost in the call to
- If I
dvc checkout --with-depsa stage that has adepwhich is a subdirectory of an import, it seems to try to checkout the entire import, not just the subdirectory- This also happens for
added assets
- This also happens for
Reproduce
How about something like this:
git init
dvc init
mkdir deps
echo 'foo' > deps/foo
echo 'bar' > deps/bar
dvc add deps
dvc remote add -d remote-upstream /tmp/dvc-remote-upstream
# dvc.yaml
stages:
dummy:
foreach:
- foo
- bar
do:
cmd: 'mkdir -p outs && echo "dep: `cat deps/${item}`" > outs/${item}'
deps:
- deps/${item}
outs:
- outs/${item}
dvc repro
git add .
git commit -m "init"
dvc push deps.dvc dvc.yaml
rm -rf .dvc/cache deps
dvc fetch --with-deps dummy@foo
dvc checkout --with-deps dummy@foo
Can then dvc import from this dummy repo to see the behavior for imported assets instead of added ones.
git init
dvc init
dvc import ../upstream outs
dvc remote add -d remote-downstream /tmp/dvc-remote-downstream
# dvc.yaml
stages:
dummy:
foreach:
- foo
- bar
do:
cmd: 'mkdir -p next && echo "out: `cat outs/${item}`" > next/${item}'
deps:
- outs/${item}
outs:
- next/${item}
dvc repro
git add .
git commit -m "init"
dvc push outs.dvc dvc.yaml
rm -rf .dvc/cache outs
dvc fetch --with-deps dummy@foo
dvc checkout --with-deps dummy@foo
Expected
deps/foo (and not deps/bar) is fetched from the remote and checked out in the workspace
Environment information
Output of dvc doctor:
$ dvc doctor
DVC version: 2.58.1 (pip)
-------------------------
Platform: Python 3.8.16 on Linux-6.2.6-76060206-generic-x86_64-with-glibc2.2.5
Subprojects:
dvc_data = 0.51.0
dvc_objects = 0.22.0
dvc_render = 0.3.1
dvc_task = 0.2.1
scmrepo = 1.0.1
Supports:
http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
s3 (s3fs = 2023.3.0, boto3 = 1.24.59),
ssh (sshfs = 2023.4.1)
Config:
Global: /home/kernel/.config/dvc
System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: overlay on overlay
Caches: local
Remotes: local
Workspace directory: overlay on overlay
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/eef9a7bf3be924c283619f6ae6f6a95e
Additional Information (if any):
$ dvc fetch --verbose dummy@foo
2023-06-05 23:49:15,320 DEBUG: v2.58.1 (pip), CPython 3.8.16 on Linux-6.2.6-76060206-generic-x86_64-with-glibc2.2.5
2023-06-05 23:49:15,320 DEBUG: command: /usr/local/bin/dvc fetch --verbose dummy@foo
2023-06-05 23:49:15,677 DEBUG: Checking if stage 'dummy@foo' is in 'dvc.yaml'
2023-06-05 23:49:15,699 DEBUG: Preparing to transfer data from '/tmp/dvc-remote' to '/tmp/tmp.AFBUiLE9S0/.dvc/cache'
2023-06-05 23:49:15,699 DEBUG: Preparing to collect status from '/tmp/tmp.AFBUiLE9S0/.dvc/cache'
2023-06-05 23:49:15,699 DEBUG: Collecting status from '/tmp/tmp.AFBUiLE9S0/.dvc/cache'
2023-06-05 23:49:15,701 DEBUG: Preparing to collect status from '/tmp/dvc-remote'
2023-06-05 23:49:15,701 DEBUG: Collecting status from '/tmp/dvc-remote'
1 file fetched
2023-06-05 23:49:15,724 DEBUG: Analytics is enabled.
2023-06-05 23:49:15,745 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpduxxmkgt']'
2023-06-05 23:49:15,746 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpduxxmkgt']'
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the add and import cases using the provided dvc.yaml stages and the dvc fetch/checkout commands with --with-deps. Start by tracing how the dependency is passed into repo.used_objs and compare fetch and checkout handling. Done means only the selected subdirectory, such as deps/foo or outs/foo, is fetched and checked out, not its sibling or the entire import.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- git, python
- Domain
- cli, devtools
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100