treeverse / treeverse/dvc

`dvc import` compatible with GitHub App Token

Open
#8,068 10 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request git
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

I haven't seen any proposal of this kind in the issues and - based on my use case - it could solve a number of problems.

Scenario:

  • you have a Data Registry (as git repo + cloud storage, e.g., AWS S3);
  • you have a Experiment Repository in which you have the code that runs experiments (and experiments use data from Data Registry);
  • you wrap this thing with CML and you use GitHub App with Access Tokens

Problem:

  • suppose you use dvc import to obtain some_data from the Data Registry (call it: github.com/username/DataRegistry)
  • it will be recorded in dvc.lock as
     deps:
       - path: some_data
         repo:
           url: git@github.com:username/DataRegistry.git
           rev_lock: af6a1feb542dc05b4d3e9c80deb50e6596876e5f
    
  • now the problem occurs: CML runs this pipeline on instance and when it tries to get the data from Data Registry remote it fails, as it cannot clone the Data Registry repository (in order to do so, it would need to use generated app token).

Proposition:

  • it would be nice if dvc import (or actually dvc pull ?) checked for DATA_REGISTRY_TOKEN env variable and updated the url "on the fly" when pulling data from the remote.

Disclaimer: I was intending on writing this some months ago, at the time the desired behaviour was not in place. I did a quick look, but did not find any mention of it.

Thanks for your effort and please ask any questions in case you need clarification!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start from the dvc import and dvc pull flows described in the issue, focusing on how the Data Registry repository URL is used during a pull. Check how the DATA_REGISTRY_TOKEN environment variable should affect GitHub App authentication, then verify that imported data can be pulled by CML without manually changing dvc.lock.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, git, github, python
Domain
authentication, cloud, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.