Strange compression ratio on large CSV file...
- Dominant language
- TypeScript
- Stars
- 14.9k
- Forks
- 1.4k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 9
Description
*It is rather a question as an issue basically, but since the repository doesn't have discussions (why actually not?) I post it here...*
Testing different compression levels on a large CSV files (with v. 1.1.0 and 1.0.9), I noticed a bit strange behavior with ratio on certain levels.
Here is an example on public file (downloadable [from here](https://excelbianalytics.com/wp/downloads-18-sample-csv-files-data-sets-for-testing-sales/), diff used for highlighting purposes only):
```diff
$ brotli -3fcv "1500000 Sales Records.csv" > /dev/null
Compressed [1500000 Sales Records.csv]: 178.511 MiB -> 58.373 MiB in 3.05 sec
$ brotli -4fcv "1500000 Sales Records.csv" > /dev/null
+ Compressed [1500000 Sales Records.csv]: 178.511 MiB -> 45.869 MiB in 5.55 sec
$ brotli -5fcv "1500000 Sales Records.csv" > /dev/null
- Compressed [1500000 Sales Records.csv]: 178.511 MiB -> 49.595 MiB in 10.15 sec
$ brotli -6fcv "1500000 Sales Records.csv" > /dev/null
- Compressed [1500000 Sales Records.csv]: 178.511 MiB -> 46.090 MiB in 15.79 sec
$ brotli -7fcv "1500000 Sales Records.csv" > /dev/null
Compressed [1500000 Sales Records.csv]: 178.511 MiB -> 41.962 MiB in 27.22 sec
```
The times looks OK, but the ratio by compression with level 4 marches to a different drummer, in fact it does that drastically and it is better than level 5 but even than level 6 too.
The question is what could cause this? And probably how we could avoid that basically slower levels 5/6 start to generate larger output or somehow heuristic force the algorithm "switch" of levels by similar circumstances?
Contributor guide
Assessment
This issue has not been assessed yet.