activeloopai / activeloopai/deeplake
[FEATURE] Skip already transferred files when restarting with hub.copy
- Langage dominant
- C++
- Étoiles
- 9.2k
- Forks
- 722
- Métriques de merge des PR
- Aucune PR mergée en 30 j
Description
## 🚨🚨 Feature Request
- Related to #1609 (closed)
### Is your feature request related to a problem?
When `hub.copy` throws an exception the transfer is terminated. Rerunning prompts to use the `overwrite=True` flag which force redownloads all of the data, ignoring the chunks that have already been downloaded.
```python
# Fails for some reason (QuotaExceeded / dropped connection / ect)
hub.copy('hub://activeloop/imagenet-val', 's3://foobar/imagenet-val', dest_creds={ ... })
# Restart prompts to run with overwrite=True
hub.copy('hub://activeloop/imagenet-val', 's3://foobar/imagenet-val', dest_creds={ ... }, overwrite=True)
```
### Description of the possible solution
When a transfer is restarted, validate existing chunks that have been transferred and skip them if they are complete.
Guide de contribution
Ouvrir le guide de contribution
Évaluation
Cette issue n'a pas encore été évaluée.