treeverse / treeverse/dvc

Extend dvc remove to work with remotes

Open
#5,340 3 comments 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

feature request p2-medium
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

Hi guys, thank you for the great tool!

We'd like to ask you for some tweaks in the process of removing data from DVC remotes.

Here is our use case: we generate some features from the raw data and store them in our DVC remotes. Unfortunately, the amount of the features exceeds our storage space (moreover, the generation process is not fixed and we experiment with it constantly, so sometimes we have multiple versions of features generated from the same source data). So, we need a way to periodically remove data from the remotes. Current one, using dvc gc together with setting and removing git tags, seems to be too fragile for our use case: it relies on precise tagging and requires careful manipulation of the tags. Additionally, it works mostly with one particular instance of git project, but we usually have several ones for different team members. Moreover, we are using the same data from several projects (we have one project working as data registry and several consumer projects), so running dvc gc without accidentally deleting some data from another project becomes quite non-trivial. Can we have the extended functionality of dvc remove instead? Something like dvc remove --remote path/to/data to remove the specified file not only from the workspace, but also from cache and remote. That way we can specify exactly what data should be removed.

Additionally it would be nice to have a way to remove only the data generated by a dvc pipeline from remote, but leave the pipeline itself untouched, so we can go back to it and regenerate data using dvc repro, but it may be a different issue.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the existing dvc remove and dvc gc entry points, then compare how remote data is selected and deleted. Use the requested dvc remove --remote path/to/data behavior as the target, while considering the separate pipeline-data case involving dvc repro; done means the intended remote deletion scope is defined and covered by tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
git
Domain
cli, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.