Force push option
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.9k
- Forks
- 1.3k
- PR merge metrics
- No merged PRs in 30d
Description
I recently experienced corrupted data when transferring large files to google cloud storage (gcs).
See discussion on Discord here.
In short, the md5 of the files at the remote was different than the md5 filename given to it by DVC. And since the md5 values at the remote were the correct ones, it was not possible to push the data one more time to get it right.
Right now there are now commands in DVC that can resolve an issue like this without losing data history.
You could do:
dvc remove data.dvc
dvc gc -w -c
dvc add data.dvc
dvc push
But this will delete all history for all files.
To solve this issue, our team had to backtrace the md5 values of the corrupted files and delete them manually from the gcs.
A "simple" solution would be to have a force option on dvc push (-f). That copies the data even if the md5 sha values are equal.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the dvc push command and its Google Cloud Storage transfer behavior described in the issue. Determine how a force option should interact with MD5-based skipping, then verify that forced pushes replace corrupted remote data without removing unrelated history.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- google-cloud, python
- Domain
- cli, cloud, data
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100