GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil does not support retry during piped downloads

Open
#504 1 comment 0 reactions 0 assignees View on GitHub
Feature Request
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

While uploads thankfully can retry since https://github.com/GoogleCloudPlatform/gsutil/issues/229 was fixed by @houglum, this does not work for downloads, which we see often break during streaming download of 230GiB files.

As we're essentially untarring the result, storing the intermediate file should not be necessary.

Memory buffering and re-starting a download from the place where the transfer stopped via a range request before continuing to pipe should be entirely possible.

Thank you for your consideration :)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.