google / google/brotli

Slow for many small string operations?

Open
#262 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
14.9k
Forks
1.4k
Avg merge
4d 20h
Merged PRs (30d)
9

Description

I've done some tests with Brotli and Zlib

I have an input file with about 30,000 strings (one per line). Each line is between 100-250 characters.

I want to compress these short strings as fast as possible. I use the `BrotliCompressBuffer` function.

With quality 0, text mode, I get about 6 K operations per second. That is, Brotli is giving me about 6000 compressed strings per second. The compression ratio is between 1.1-1.3.

For reference, Zlib (fast mode) gets me 94 K operations per second with a compression ratio between 1.0-1.2.

Is this expected? Brotli seems to be quite a bit slower here. The marginal better compression ratio isn't worth the trade off for me right now.

I have another benchmark where I generate low entropy text strings and counter the bytes going in and out (compressed bytes). With Brotli I get about 21 MiB in and 2 MiB out per second, 5.8 K op/s. Ratio around 11. With Zlib I get about 690 MiB in and 151 MiB out, 190 K op/s. Ratio around 4.5.

Despite Brotli having a more than 2x compression ratio here it's taking way too long to set up? From the http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf paper we can see listings of throughput speeds that outclass Zlib. However, I can clearly demonstrate that his is not the case here. Is the algorithm old (I was using the 0.2 release) or slow in this regard, or does it simply have a higher overhead over Zlib and this is why it takes so much longer to do many small compression operations? I noticed that the `BrotliCompressor` does heap allocations, how much of an issue is that? Is it possible to get a compressor that does not allocate additional memory?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.