Use better gzip/deflate
- Dominant language
- Go
- Stars
- 7.4k
- Forks
- 643
- Avg merge
- 14h 5m
- Merged PRs (30d)
- 2
Description
The standard library is more interested in maintaining status quo than improving compression.
I maintain a [compression package](https://github.com/klauspost/compres) that seems more suited to Hertz. It plugs in without any modifications, by simply replacing imports.
Below is a comparison between my package and the standard library, using [4GB of mixed web content](https://files.klauspost.com/compress/webdevdata.org-2015-01-07-4GB-subset.7z):
| out | level | insize | outsize | millis | mb/s | Files/s | MB/s | Reduction | Rel speed | Smaller | Gain |
|:-----:|------:|-----------:|-----------:|-------:|-------:|---------|--------|-----------|-----------|---------|---------|
| gzstd | 1 | 4014735833 | 862232427 | 20605 | 185.81 | 2617 | 185.82 | 78.52% | 1.00x | 0.000% | 0.000% |
| gzkp | 1 | 4014735833 | 866657631 | 13860 | 276.24 | 3891 | 276.24 | 78.41% | 1.49x | -0.513% | -0.110% |
| | | | | | | | | | | | |
| gzstd | 2 | 4014735833 | 813642417 | 26380 | 145.14 | 2044 | 145.14 | 79.73% | 1.00x | 0.000% | 0.000% |
| gzkp | 2 | 4014735833 | 842584206 | 15322 | 249.87 | 3520 | 249.89 | 79.01% | 1.72x | -3.557% | -0.721% |
| | | | | | | | | | | | |
| gzstd | 3 | 4014735833 | 800582015 | 27842 | 137.51 | 1937 | 137.52 | 80.06% | 1.00x | 0.000% | 0.000% |
| gzkp | 3 | 4014735833 | 823209525 | 17003 | 225.17 | 3172 | 225.18 | 79.50% | 1.64x | -2.826% | -0.564% |
| | | | | | | | | | | | |
| gzstd | 4 | 4014735833 | 765294480 | 34710 | 110.31 | 1554 | 110.31 | 80.94% | 1.00x | 0.000% | 0.000% |
| gzkp | 4 | 4014735833 | 791607893 | 19778 | 193.58 | 2727 | 193.59 | 80.28% | 1.75x | -3.438% | -0.655% |
| | | | | | | | | | | | |
| gzstd | 5 | 4014735833 | 744808940 | 42672 | 89.72 | 1264 | 89.73 | 81.45% | 1.00x | 0.000% | 0.000% |
| gzkp | 5 | 4014735833 | 764683084 | 22471 | 170.38 | 2400 | 170.39 | 80.95% | 1.90x | -2.668% | -0.495% |
| | | | | | | | | | | | |
| gzstd | 6 | 4014735833 | 735318526 | 51993 | 73.64 | 1037 | 73.64 | 81.68% | 1.00x | 0.000% | 0.000% |
| gzkp | 6 | 4014735833 | 753831382 | 24787 | 154.46 | 2176 | 154.47 | 81.22% | 2.10x | -2.518% | -0.461% |
| | | | | | | | | | | | |
| gzstd | 7 | 4014735833 | 733144257 | 60588 | 63.19 | 890 | 63.19 | 81.74% | 1.00x | 0.000% | 0.000% |
| gzkp | 7 | 4014735833 | 746309058 | 34562 | 110.78 | 1560 | 110.78 | 81.41% | 1.75x | -1.796% | -0.328% |
| | | | | | | | | | | | |
| gzstd | 8 | 4014735833 | 730369312 | 77883 | 49.16 | 692 | 49.16 | 81.81% | 1.00x | 0.000% | 0.000% |
| gzkp | 8 | 4014735833 | 735523067 | 43270 | 88.48 | 1246 | 88.49 | 81.68% | 1.80x | -0.706% | -0.128% |
| | | | | | | | | | | | |
| gzstd | 9 | 4014735833 | 729905331 | 94517 | 40.51 | 571 | 40.51 | 81.82% | 1.00x | 0.000% | 0.000% |
| gzkp | 9 | 4014735833 | 727900632 | 94305 | 40.6 | 572 | 40.60 | 81.87% | 1.00x | 0.275% | 0.050% |
| | | | | | | | | | | | |
| gzstd | -2 | 4014735833 | 2617482950 | 12742 | 300.48 | 4232 | 300.48 | 34.80% | 1.00x | 0.000% | 0.000% |
| gzkp | -2 | 4014735833 | 2604521759 | 9281 | 412.49 | 5810 | 412.54 | 35.13% | 1.37x | 0.495% | 0.323% |
The standard library is poorly balanced, so levels 6-8 have very little difference. This is more spread out so the actual level reflects an increase in compression. Here is the compression/speed tradeoff visualized:

The package uses no cgo or "unsafe" code.
TLDR; With level 6 (default in your setup), you get 2.1x the throughput at 2.7% more output on the wire, with better options to tweak speed/compression.
**Describe the solution you'd like**
Replace gzip/flate imports with [corresponding packages](https://github.com/klauspost/compress#deflate-usage).
**Additional context**
The package also provides [stateless compression modes](https://github.com/klauspost/compress#stateless-compression) which can be useful for long-running requests with little data throughput.
Each Write will be compressed immediately without any state being retained between calls. I don't know if this has any use here.
Contributor guide
Assessment
This issue has not been assessed yet.