google / google/brotli

Strange compression ratio on large CSV file...

Open
#1,071 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
TypeScript
Stars
14.9k
Forks
1.4k
Avg merge
4d 20h
Merged PRs (30d)
9

Description

*It is rather a question as an issue basically, but since the repository doesn't have discussions (why actually not?) I post it here...*

Testing different compression levels on a large CSV files (with v. 1.1.0 and 1.0.9), I noticed a bit strange behavior with ratio on certain levels.
Here is an example on public file (downloadable [from here](https://excelbianalytics.com/wp/downloads-18-sample-csv-files-data-sets-for-testing-sales/), diff used for highlighting purposes only):
```diff
$ brotli -3fcv "1500000 Sales Records.csv" > /dev/null
Compressed [1500000 Sales Records.csv]: 178.511 MiB -> 58.373 MiB in 3.05 sec

$ brotli -4fcv "1500000 Sales Records.csv" > /dev/null
+ Compressed [1500000 Sales Records.csv]: 178.511 MiB -> 45.869 MiB in 5.55 sec

$ brotli -5fcv "1500000 Sales Records.csv" > /dev/null
- Compressed [1500000 Sales Records.csv]: 178.511 MiB -> 49.595 MiB in 10.15 sec

$ brotli -6fcv "1500000 Sales Records.csv" > /dev/null
- Compressed [1500000 Sales Records.csv]: 178.511 MiB -> 46.090 MiB in 15.79 sec

$ brotli -7fcv "1500000 Sales Records.csv" > /dev/null
Compressed [1500000 Sales Records.csv]: 178.511 MiB -> 41.962 MiB in 27.22 sec
```
The times looks OK, but the ratio by compression with level 4 marches to a different drummer, in fact it does that drastically and it is better than level 5 but even than level 6 too.

The question is what could cause this? And probably how we could avoid that basically slower levels 5/6 start to generate larger output or somehow heuristic force the algorithm "switch" of levels by similar circumstances?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.