GoogleCloudPlatform / GoogleCloudPlatform/gsutil

gsutil -m cp -I doesn't support parallel uploads

Open
#491 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Hi all,

My lab is using gsutil for uploading and downloading chunked objects that comprise a three dimensional image. Each object is relatively small (< 1MB) but collectively can add up to terabytes.

We were investigating using gsutil -m to make our downloads go faster and were using a construction like follows:

1. Generate the filenames that comprise an image cutout.
2. Pipe them into gsutil like so:

```
gsutil_upload_cmd = "gsutil -m {headers} cp -I {compress} gs://{cloudpath}".format(
headers=" ".join(headers),
compress=('-Z' if compress else ''),
cloudpath=cloudpath,
)

print(gsutil_upload_cmd)

gcs_pipe = subprocess.Popen([gsutil_upload_cmd],
stdin=subprocess.PIPE,
stdout=sys.stdout,
shell=True
)

# shoves filenames into pipe stdin, waits for process to execute, and terminates
# returns stdout
gcs_pipe.communicate(input="\n".join(filenames))
```
In our experiments we found that -m does work, but gsutil appears to be reading in a single file name at a time and executing in parallel on that single chunk (if possible). Would it be possible for gsutil to read in multiple lines at a time or if that's too risky a change, to provide some kind of other delimiter that would allow us to feed in an arbitrary number of items at a time?

Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.