GoogleCloudPlatform / GoogleCloudPlatform/gsutil
Set compression level for `-Z`
- Dominant language
- Python
- Stars
- 918
- Forks
- 335
- PR merge metrics
- No merged PRs in 30d
Description
Hi Team,
In our tests the uploads using `-Z` take a really long time (in one test, 110 min vs 7 min for a "normal" upload). We also tried compressing the same file locally first with `gzip`, and pipe the output to `gsutil`. It appears we can achieve fast compression+upload (~6 min in total) if we set the compression level to 1 (as in, `gzip -1c file | gsutil cp - gs://...`), and the resultant file size is only 30% higher than with the default option.
Would there be a possibility to add a native compression level option to `gsutil`? This way, we could avoid an additional step of piping the input from `gzip` (as well as piping the output through it when downloading the compressed file).
Thanks
Contributor guide
Assessment
This issue has not been assessed yet.