treeverse / treeverse/dvc

index: repo imports are not handled correctly in repo index

Open
#8,571 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A: data-management
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Repo import entries are currently generated according to the local repo's output entry. So on data_index.load(), any files/subdirectories in a repo import are generated using the output md5s, and they will have the local repo's default remote set (and only use the output md5 hash infos, whether the imported file is a dvc object or a git file).

On data_index.load(), the repo import index entries should be generated properly: git files should have fs & path entries that refer to the source repo's repofs, and dvc objects should be using the proper source repo remotes (where different files within the imported dir may all have unique remotes due to outs: remote: usage)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at data_index.load() and trace how repo import entries are generated from the local repo's output entry. Compare handling of git files with source repo repofs and DVC objects with source repo remotes, including per-file outs: remote values. Done means imported index entries use the source repository's paths, filesystems, and remotes rather than the local output metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
data
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.