treeverse / treeverse/dvc

fetch/push/status: not handling config from other revisions

Open
#9,754 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

A: data-sync bug p2-medium
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

When branches have wildly different remote setups, those configs are not taken into account during fetch/push/status -c --all-tags/branches/etc

Example:

#!/bin/bash

set -e
set -x

rm -rf mytest
mkdir mytest
cd mytest

mkdir remote1
mkdir remote2
remote1="$(pwd)/remote1"
remote2="$(pwd)/remote2"

mkdir repo
cd repo
git init
dvc init
git commit -m "init"
git branch branch1
git branch branch2

git checkout branch1
echo foo > foo
dvc add foo
dvc remote add -d myremote1 $remote1
dvc push
git add .gitignore foo.dvc .dvc/config
git commit -m "foo"

git checkout branch2
echo bar > bar
dvc add bar
dvc remote add -d myremote2 $remote2
dvc push
git add .gitignore bar.dvc .dvc/config
git commit -m "bar"

git checkout main
rm -rf .dvc/cache
dvc fetch --all-branches
tree .dvc/cache  # will show 0 files

Studio uses real git checkout to collect objects and has been doing that for years as a workaround, but I couldn't find an issue in dvc yet.

To fix this we should make config part of Index(same as stages, outs, etc are, don't confuse with DataIndex) and use it to build Index.data. This is the easiest to do in dvc fetch because it is using Index.data already, but might require temporary workarounds for push/status -c like manually triggering config reloading in brancher or something.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the existing dvc fetch implementation and its use of Index.data, then inspect how brancher reloads configuration for push and status -c. The remaining work is to account for config from other revisions in push and status -c, considering the dependency on issue #9333, and verify the multi-branch example.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
cli, tooling
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.