GoogleCloudPlatform / GoogleCloudPlatform/gsutil
gsutil -m cp -I doesn't support parallel uploads
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
Hi all,
My lab is using gsutil for uploading and downloading chunked objects that comprise a three dimensional image. Each object is relatively small (< 1MB) but collectively can add up to terabytes.
We were investigating using gsutil -m to make our downloads go faster and were using a construction like follows:
1. Generate the filenames that comprise an image cutout.
2. Pipe them into gsutil like so:
```
gsutil_upload_cmd = "gsutil -m {headers} cp -I {compress} gs://{cloudpath}".format(
headers=" ".join(headers),
compress=('-Z' if compress else ''),
cloudpath=cloudpath,
)
print(gsutil_upload_cmd)
gcs_pipe = subprocess.Popen([gsutil_upload_cmd],
stdin=subprocess.PIPE,
stdout=sys.stdout,
shell=True
)
# shoves filenames into pipe stdin, waits for process to execute, and terminates
# returns stdout
gcs_pipe.communicate(input="\n".join(filenames))
```
In our experiments we found that -m does work, but gsutil appears to be reading in a single file name at a time and executing in parallel on that single chunk (if possible). Would it be possible for gsutil to read in multiple lines at a time or if that's too risky a change, to provide some kind of other delimiter that would allow us to feed in an arbitrary number of items at a time?
Thank you!
Contributor guide
Assessment
This issue has not been assessed yet.