treeverse / treeverse/dvc

Force push option

Open
#7,268 11 comments 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

A: data-sync feature request p2-medium
Dominant language
Python
Stars
15.9k
Forks
1.3k
PR merge metrics
No merged PRs in 30d

Description

I recently experienced corrupted data when transferring large files to google cloud storage (gcs).

See discussion on Discord here.

In short, the md5 of the files at the remote was different than the md5 filename given to it by DVC. And since the md5 values at the remote were the correct ones, it was not possible to push the data one more time to get it right.

Right now there are now commands in DVC that can resolve an issue like this without losing data history.

You could do:

dvc remove data.dvc
dvc gc -w -c
dvc add data.dvc
dvc push

But this will delete all history for all files.

To solve this issue, our team had to backtrace the md5 values of the corrupted files and delete them manually from the gcs.

A "simple" solution would be to have a force option on dvc push (-f). That copies the data even if the md5 sha values are equal.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the dvc push command and its Google Cloud Storage transfer behavior described in the issue. Determine how a force option should interact with MD5-based skipping, then verify that forced pushes replace corrupted remote data without removing unrelated history.

Written by the indexing model from the issue text.

Assessment

Tech stack
google-cloud, python
Domain
cli, cloud, data
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.