treeverse / treeverse/dvc

pull: KeyError crash (or silently skipped target) when mixing .dvc-file targets with granular paths inside a tracked directory

Open
#11,075 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Bug Report

Description

dvc pull with a target list that mixes a .dvc-file target with granular paths inside a tracked directory misbehaves in two ways, depending on workspace state:

  1. Crash: if the tracked directory contains any untracked (drifted) file, pull dies with ERROR: unexpected error - ('datadir', 'f1.txt') — an uncaught KeyError from checkout's _check_can_delete.
  2. Silent skip: in a clean fresh-clone state (no workspace, no cache), the same command succeeds (exit 0) but only materializes the .dvc-file target; the granular target is never checked out.

Both target forms are documented as supported in the pull command reference ("paths to tracked files or directories (including paths inside tracked directories), .dvc files, and stage names"), and each form works correctly on its own — including a single invocation whose targets are all data paths, which is the workaround.

Reproduce

#!/usr/bin/env bash
set -uo pipefail
cd "$(mktemp -d)"
mkdir remote repo && cd repo
git init -q && git config user.email r@e.pro && git config user.name repro
dvc init -q
dvc remote add -d localremote ../remote
mkdir datadir
printf one > datadir/f1.txt; printf two > datadir/f2.txt; printf three > datadir/f3.txt
printf single > single.csv
dvc add datadir single.csv
git add -A && git commit -qm init
dvc push -q

echo "=== Variant 1: fresh-clone state -> granular target silently skipped ==="
rm -rf datadir single.csv .dvc/cache
dvc pull single.csv.dvc datadir/f1.txt        # exit 0, "1 file fetched and 1 file added"
ls datadir/f1.txt 2>/dev/null || echo "BUG: datadir/f1.txt was not checked out"

dvc pull -q   # restore everything

echo "=== Variant 2: one untracked file inside the tracked dir -> crash ==="
printf extra > datadir/extra-drift.txt
dvc pull single.csv.dvc datadir/f1.txt        # ERROR: unexpected error - ('datadir', 'f1.txt')

echo "=== Variant 3 (control): same drift, data-path targets only -> works ==="
dvc pull single.csv datadir/f1.txt            # "Everything is up to date."

Expected

Either the mixed target list works (both targets checked out, like variant 3), or a proper error message — not an internal KeyError and not a silent partial result.

Traceback (variant 2)

Traceback (most recent call last):
  ...
  File ".../dvc/repo/pull.py", line 42, in pull
  File ".../dvc/repo/checkout.py", line 160, in checkout
    _check_can_delete(diff.files_delete, new, self.root_dir, self.fs)
  File ".../dvc/repo/checkout.py", line 91, in _check_can_delete
    cache_fs, cache_path = index.storage_map.get_cache(entry)
  File ".../dvc_data/index/index.py", line 519, in get_cache
  File ".../dvc_data/index/index.py", line 508, in get_storage
    info = self[entry.key]
  File ".../dvc_data/index/index.py", line 446, in __getitem__
KeyError: ('datadir', 'f1.txt')

Analysis

_check_can_delete anticipates the "entry known but no cache storage" case — it wraps the lookup in try/except ValueError (catching StorageKeyError). But when targets span different artifacts, the filtered index views seem to produce files_delete entries whose keys are entirely absent from the (filtered) storage_map, so StorageMapping.__getitem__ raises a bare KeyError, which escapes to the generic "unexpected error" handler.

Environment

DVC version: 3.67.1 (pip)
-------------------------
Platform: Python 3.12.12 on Linux-7.0.12-101.fc43.x86_64-x86_64-with-glibc2.42
Subprojects:
        dvc_data = 3.18.3
        dvc_objects = 5.2.0
        dvc_render = 1.0.2
        dvc_task = 0.40.2
        scmrepo = 3.6.2
Supports: http, https, s3 (s3fs = 2026.4.0)

Originally hit with an S3 (Cloudflare R2) remote pulling two standalone-file artifacts plus 10 granular paths inside a 70-file directory artifact; the local-remote repro above shows the remote type is irrelevant.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with dvc/repo/pull.py and dvc/repo/checkout.py, especially checkout and _check_can_delete, then reproduce the mixed-target commands from the issue. Compare the filtered index and storage-map behavior for .dvc-file and granular targets. Done means both targets are checked out without a KeyError or silent partial result, with regression coverage for the reported variants.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.