cloudwego / cloudwego/hertz

Use better gzip/deflate

Open
#343 4 comments 2 reactions 1 assignee Claimed by @errocks View on GitHub
enhancement NeedsDecision
Dominant language
Go
Stars
7.4k
Forks
643
Avg merge
14h 5m
Merged PRs (30d)
2

Description

The standard library is more interested in maintaining status quo than improving compression.

I maintain a [compression package](https://github.com/klauspost/compres) that seems more suited to Hertz. It plugs in without any modifications, by simply replacing imports.

Below is a comparison between my package and the standard library, using [4GB of mixed web content](https://files.klauspost.com/compress/webdevdata.org-2015-01-07-4GB-subset.7z):

| out | level | insize | outsize | millis | mb/s | Files/s | MB/s | Reduction | Rel speed | Smaller | Gain |
|:-----:|------:|-----------:|-----------:|-------:|-------:|---------|--------|-----------|-----------|---------|---------|
| gzstd | 1 | 4014735833 | 862232427 | 20605 | 185.81 | 2617 | 185.82 | 78.52% | 1.00x | 0.000% | 0.000% |
| gzkp | 1 | 4014735833 | 866657631 | 13860 | 276.24 | 3891 | 276.24 | 78.41% | 1.49x | -0.513% | -0.110% |
| | | | | | | | | | | | |
| gzstd | 2 | 4014735833 | 813642417 | 26380 | 145.14 | 2044 | 145.14 | 79.73% | 1.00x | 0.000% | 0.000% |
| gzkp | 2 | 4014735833 | 842584206 | 15322 | 249.87 | 3520 | 249.89 | 79.01% | 1.72x | -3.557% | -0.721% |
| | | | | | | | | | | | |
| gzstd | 3 | 4014735833 | 800582015 | 27842 | 137.51 | 1937 | 137.52 | 80.06% | 1.00x | 0.000% | 0.000% |
| gzkp | 3 | 4014735833 | 823209525 | 17003 | 225.17 | 3172 | 225.18 | 79.50% | 1.64x | -2.826% | -0.564% |
| | | | | | | | | | | | |
| gzstd | 4 | 4014735833 | 765294480 | 34710 | 110.31 | 1554 | 110.31 | 80.94% | 1.00x | 0.000% | 0.000% |
| gzkp | 4 | 4014735833 | 791607893 | 19778 | 193.58 | 2727 | 193.59 | 80.28% | 1.75x | -3.438% | -0.655% |
| | | | | | | | | | | | |
| gzstd | 5 | 4014735833 | 744808940 | 42672 | 89.72 | 1264 | 89.73 | 81.45% | 1.00x | 0.000% | 0.000% |
| gzkp | 5 | 4014735833 | 764683084 | 22471 | 170.38 | 2400 | 170.39 | 80.95% | 1.90x | -2.668% | -0.495% |
| | | | | | | | | | | | |
| gzstd | 6 | 4014735833 | 735318526 | 51993 | 73.64 | 1037 | 73.64 | 81.68% | 1.00x | 0.000% | 0.000% |
| gzkp | 6 | 4014735833 | 753831382 | 24787 | 154.46 | 2176 | 154.47 | 81.22% | 2.10x | -2.518% | -0.461% |
| | | | | | | | | | | | |
| gzstd | 7 | 4014735833 | 733144257 | 60588 | 63.19 | 890 | 63.19 | 81.74% | 1.00x | 0.000% | 0.000% |
| gzkp | 7 | 4014735833 | 746309058 | 34562 | 110.78 | 1560 | 110.78 | 81.41% | 1.75x | -1.796% | -0.328% |
| | | | | | | | | | | | |
| gzstd | 8 | 4014735833 | 730369312 | 77883 | 49.16 | 692 | 49.16 | 81.81% | 1.00x | 0.000% | 0.000% |
| gzkp | 8 | 4014735833 | 735523067 | 43270 | 88.48 | 1246 | 88.49 | 81.68% | 1.80x | -0.706% | -0.128% |
| | | | | | | | | | | | |
| gzstd | 9 | 4014735833 | 729905331 | 94517 | 40.51 | 571 | 40.51 | 81.82% | 1.00x | 0.000% | 0.000% |
| gzkp | 9 | 4014735833 | 727900632 | 94305 | 40.6 | 572 | 40.60 | 81.87% | 1.00x | 0.275% | 0.050% |
| | | | | | | | | | | | |
| gzstd | -2 | 4014735833 | 2617482950 | 12742 | 300.48 | 4232 | 300.48 | 34.80% | 1.00x | 0.000% | 0.000% |
| gzkp | -2 | 4014735833 | 2604521759 | 9281 | 412.49 | 5810 | 412.54 | 35.13% | 1.37x | 0.495% | 0.323% |

The standard library is poorly balanced, so levels 6-8 have very little difference. This is more spread out so the actual level reflects an increase in compression. Here is the compression/speed tradeoff visualized:

![chart](https://user-images.githubusercontent.com/5663952/199524248-54720258-c114-4424-87fa-ffb587de5120.png)

The package uses no cgo or "unsafe" code.

TLDR; With level 6 (default in your setup), you get 2.1x the throughput at 2.7% more output on the wire, with better options to tweak speed/compression.

**Describe the solution you'd like**

Replace gzip/flate imports with [corresponding packages](https://github.com/klauspost/compress#deflate-usage).

**Additional context**

The package also provides [stateless compression modes](https://github.com/klauspost/compress#stateless-compression) which can be useful for long-running requests with little data throughput.

Each Write will be compressed immediately without any state being retained between calls. I don't know if this has any use here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.