activeloopai / activeloopai/deeplake

[FEATURE] Skip already transferred files when restarting with hub.copy

未關閉
#1,617 2 則留言 1 個 reaction 已指派 0 人 在 GitHub 檢視
enhancement
主要語言
C++
星號
9.2k
分支
722
PR 合併指標
30 天內沒有已合併 PR

描述

## 🚨🚨 Feature Request

- Related to #1609 (closed)

### Is your feature request related to a problem?

When `hub.copy` throws an exception the transfer is terminated. Rerunning prompts to use the `overwrite=True` flag which force redownloads all of the data, ignoring the chunks that have already been downloaded.

```python
# Fails for some reason (QuotaExceeded / dropped connection / ect)
hub.copy('hub://activeloop/imagenet-val', 's3://foobar/imagenet-val', dest_creds={ ... })

# Restart prompts to run with overwrite=True
hub.copy('hub://activeloop/imagenet-val', 's3://foobar/imagenet-val', dest_creds={ ... }, overwrite=True)
```

### Description of the possible solution

When a transfer is restarted, validate existing chunks that have been transferred and skip them if they are complete.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。