`dvc import` compatible with GitHub App Token
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.9k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
I haven't seen any proposal of this kind in the issues and - based on my use case - it could solve a number of problems.
Scenario:
- you have a Data Registry (as git repo + cloud storage, e.g., AWS S3);
- you have a Experiment Repository in which you have the code that runs experiments (and experiments use data from Data Registry);
- you wrap this thing with CML and you use GitHub App with Access Tokens
Problem:
- suppose you use
dvc importto obtainsome_datafrom the Data Registry (call it:github.com/username/DataRegistry) - it will be recorded in
dvc.lockasdeps: - path: some_data repo: url: git@github.com:username/DataRegistry.git rev_lock: af6a1feb542dc05b4d3e9c80deb50e6596876e5f - now the problem occurs: CML runs this pipeline on instance and when it tries to get the data from Data Registry remote it fails, as it cannot clone the Data Registry repository (in order to do so, it would need to use generated app token).
Proposition:
- it would be nice if
dvc import(or actuallydvc pull?) checked forDATA_REGISTRY_TOKENenv variable and updated the url "on the fly" when pulling data from the remote.
Disclaimer: I was intending on writing this some months ago, at the time the desired behaviour was not in place. I did a quick look, but did not find any mention of it.
Thanks for your effort and please ask any questions in case you need clarification!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start from the dvc import and dvc pull flows described in the issue, focusing on how the Data Registry repository URL is used during a pull. Check how the DATA_REGISTRY_TOKEN environment variable should affect GitHub App authentication, then verify that imported data can be pulled by CML without manually changing dvc.lock.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, git, github, python
- Domain
- authentication, cloud, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100