treeverse / treeverse/dvc

checkout and fetch: .dvc stages are not resolved correctly when using --with-deps

Open
#9,543 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A: data-management A: data-sync bug
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Bug Report

Description

  • When I dvc fetch a subdirectory of an imported asset (tracked by .dvc file), I see a "N files fetched" message. However, doing dvc checkout of the same subdir has no effect (nothing checked out in workspace).
    • As might be expected, I see the same behavior if I do dvc pull instead.
    • If the asset is an added asset instead of an imported one, all files in the asset are fetched
  • Related, when I dvc fetch --with-deps a stage that has a dep which is a subdirectory of an import, it doesn't fetch the data for that dependency.
    • I traced down the code and it looks like the dependency gets lost in the call to repo.used_objs()
    • This happens both in versions 2.58.2 and 2.53.0.
    • If I drop down to 2.52.0 then fetch goes back to the old behavior of pulling too much data 😂 (see #6124 and #9246)
  • If I dvc checkout --with-deps a stage that has a dep which is a subdirectory of an import, it seems to try to checkout the entire import, not just the subdirectory
    • This also happens for added assets
Reproduce

How about something like this:

git init
dvc init
mkdir deps
echo 'foo' > deps/foo
echo 'bar' > deps/bar
dvc add deps
dvc remote add -d remote-upstream /tmp/dvc-remote-upstream
# dvc.yaml
stages:
  dummy:
    foreach:
      - foo
      - bar
    do:
      cmd: 'mkdir -p outs && echo "dep: `cat deps/${item}`" > outs/${item}'
      deps:
        - deps/${item}
      outs:
        - outs/${item}
dvc repro
git add .
git commit -m "init"
dvc push deps.dvc dvc.yaml

rm -rf .dvc/cache deps
dvc fetch --with-deps dummy@foo
dvc checkout --with-deps dummy@foo

Can then dvc import from this dummy repo to see the behavior for imported assets instead of added ones.

git init
dvc init
dvc import ../upstream outs
dvc remote add -d remote-downstream /tmp/dvc-remote-downstream
# dvc.yaml 
stages:
  dummy:
    foreach:
      - foo
      - bar
    do:
      cmd: 'mkdir -p next && echo "out: `cat outs/${item}`" > next/${item}'
      deps:
        - outs/${item}
      outs:
        - next/${item}
dvc repro
git add .
git commit -m "init"
dvc push outs.dvc dvc.yaml

rm -rf .dvc/cache outs
dvc fetch --with-deps dummy@foo
dvc checkout --with-deps dummy@foo
Expected

deps/foo (and not deps/bar) is fetched from the remote and checked out in the workspace

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 2.58.1 (pip)
-------------------------
Platform: Python 3.8.16 on Linux-6.2.6-76060206-generic-x86_64-with-glibc2.2.5
Subprojects:
        dvc_data = 0.51.0
        dvc_objects = 0.22.0
        dvc_render = 0.3.1
        dvc_task = 0.2.1
        scmrepo = 1.0.1
Supports:
        http (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        https (aiohttp = 3.8.4, aiohttp-retry = 2.8.3),
        s3 (s3fs = 2023.3.0, boto3 = 1.24.59),
        ssh (sshfs = 2023.4.1)
Config:
        Global: /home/kernel/.config/dvc
        System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: overlay on overlay
Caches: local
Remotes: local
Workspace directory: overlay on overlay
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/eef9a7bf3be924c283619f6ae6f6a95e

Additional Information (if any):

$ dvc fetch --verbose dummy@foo
2023-06-05 23:49:15,320 DEBUG: v2.58.1 (pip), CPython 3.8.16 on Linux-6.2.6-76060206-generic-x86_64-with-glibc2.2.5
2023-06-05 23:49:15,320 DEBUG: command: /usr/local/bin/dvc fetch --verbose dummy@foo
2023-06-05 23:49:15,677 DEBUG: Checking if stage 'dummy@foo' is in 'dvc.yaml'
2023-06-05 23:49:15,699 DEBUG: Preparing to transfer data from '/tmp/dvc-remote' to '/tmp/tmp.AFBUiLE9S0/.dvc/cache'
2023-06-05 23:49:15,699 DEBUG: Preparing to collect status from '/tmp/tmp.AFBUiLE9S0/.dvc/cache'
2023-06-05 23:49:15,699 DEBUG: Collecting status from '/tmp/tmp.AFBUiLE9S0/.dvc/cache'
2023-06-05 23:49:15,701 DEBUG: Preparing to collect status from '/tmp/dvc-remote'          
2023-06-05 23:49:15,701 DEBUG: Collecting status from '/tmp/dvc-remote'
1 file fetched                                                                             
2023-06-05 23:49:15,724 DEBUG: Analytics is enabled.                                       
2023-06-05 23:49:15,745 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmpduxxmkgt']'
2023-06-05 23:49:15,746 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmpduxxmkgt']'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the add and import cases using the provided dvc.yaml stages and the dvc fetch/checkout commands with --with-deps. Start by tracing how the dependency is passed into repo.used_objs and compare fetch and checkout handling. Done means only the selected subdirectory, such as deps/foo or outs/foo, is fetched and checked out, not its sibling or the entire import.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
cli, devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.