activeloopai / activeloopai/deeplake

[FEATURE] Skip already transferred files when restarting with hub.copy

Ouverte
#1,617 2 commentaires 1 réaction 0 personnes assignées Voir sur GitHub
enhancement
Langage dominant
C++
Étoiles
9.2k
Forks
722
Métriques de merge des PR
Aucune PR mergée en 30 j

Description

## 🚨🚨 Feature Request

- Related to #1609 (closed)

### Is your feature request related to a problem?

When `hub.copy` throws an exception the transfer is terminated. Rerunning prompts to use the `overwrite=True` flag which force redownloads all of the data, ignoring the chunks that have already been downloaded.

```python
# Fails for some reason (QuotaExceeded / dropped connection / ect)
hub.copy('hub://activeloop/imagenet-val', 's3://foobar/imagenet-val', dest_creds={ ... })

# Restart prompts to run with overwrite=True
hub.copy('hub://activeloop/imagenet-val', 's3://foobar/imagenet-val', dest_creds={ ... }, overwrite=True)
```

### Description of the possible solution

When a transfer is restarted, validate existing chunks that have been transferred and skip them if they are complete.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.