activeloopai / activeloopai/deeplake

[FEATURE] Skip already transferred files when restarting with hub.copy

Open
#1,617 2 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
C++
Stars
9.2k
Forks
722
PR merge metrics
No merged PRs in 30d

Description

## 🚨🚨 Feature Request

- Related to #1609 (closed)

### Is your feature request related to a problem?

When `hub.copy` throws an exception the transfer is terminated. Rerunning prompts to use the `overwrite=True` flag which force redownloads all of the data, ignoring the chunks that have already been downloaded.

```python
# Fails for some reason (QuotaExceeded / dropped connection / ect)
hub.copy('hub://activeloop/imagenet-val', 's3://foobar/imagenet-val', dest_creds={ ... })

# Restart prompts to run with overwrite=True
hub.copy('hub://activeloop/imagenet-val', 's3://foobar/imagenet-val', dest_creds={ ... }, overwrite=True)
```

### Description of the possible solution

When a transfer is restarted, validate existing chunks that have been transferred and skip them if they are complete.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.