Slow for many small string operations?
- Dominant language
- TypeScript
- Stars
- 14.9k
- Forks
- 1.4k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 9
Description
I've done some tests with Brotli and Zlib
I have an input file with about 30,000 strings (one per line). Each line is between 100-250 characters.
I want to compress these short strings as fast as possible. I use the `BrotliCompressBuffer` function.
With quality 0, text mode, I get about 6 K operations per second. That is, Brotli is giving me about 6000 compressed strings per second. The compression ratio is between 1.1-1.3.
For reference, Zlib (fast mode) gets me 94 K operations per second with a compression ratio between 1.0-1.2.
Is this expected? Brotli seems to be quite a bit slower here. The marginal better compression ratio isn't worth the trade off for me right now.
I have another benchmark where I generate low entropy text strings and counter the bytes going in and out (compressed bytes). With Brotli I get about 21 MiB in and 2 MiB out per second, 5.8 K op/s. Ratio around 11. With Zlib I get about 690 MiB in and 151 MiB out, 190 K op/s. Ratio around 4.5.
Despite Brotli having a more than 2x compression ratio here it's taking way too long to set up? From the http://www.gstatic.com/b/brotlidocs/brotli-2015-09-22.pdf paper we can see listings of throughput speeds that outclass Zlib. However, I can clearly demonstrate that his is not the case here. Is the algorithm old (I was using the 0.2 release) or slow in this regard, or does it simply have a higher overhead over Zlib and this is why it takes so much longer to do many small compression operations? I noticed that the `BrotliCompressor` does heap allocations, how much of an issue is that? Is it possible to get a compressor that does not allocate additional memory?
Contributor guide
Assessment
This issue has not been assessed yet.