GoogleCloudPlatform / GoogleCloudPlatform/gsutil

Set compression level for `-Z`

Open
#508 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
918
Forks
335
PR merge metrics
No merged PRs in 30d

Description

Hi Team,

In our tests the uploads using `-Z` take a really long time (in one test, 110 min vs 7 min for a "normal" upload). We also tried compressing the same file locally first with `gzip`, and pipe the output to `gsutil`. It appears we can achieve fast compression+upload (~6 min in total) if we set the compression level to 1 (as in, `gzip -1c file | gsutil cp - gs://...`), and the resultant file size is only 30% higher than with the default option.

Would there be a possibility to add a native compression level option to `gsutil`? This way, we could avoid an additional step of piping the input from `gzip` (as well as piping the output through it when downloading the compressed file).

Thanks

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.