How to achieve better compression on 1MB block by re-using compression context
- Dominant language
- C
- Stars
- 27.9k
- Forks
- 2.6k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 8
Description
I am compressing 1G JSON file, with zstd command, and getting 27.55 MB file (2.75% compression ratio - very impressive!!!!).
When I am compressing same file, breaking input into 1MB blocks, and compressing each block individually into frame, I am getting 41.7MB file, around 50% worse compression ratio than zstd command line tool.
Increasing block size to 10MB make compression ratio go closer to original but still far.
I suspect zstd utility is building dictionary on very large data sample, much larger than 10MB, and it makes much better compression ratio.
Obviously you need to reuse compression context, in hope that dictionary will be reused between invocations of ZSTD_compress.
Code (v1.5.2) explicitly saying it is not going to work:
_Note : re-using context is just a speed / resource optimization.
It doesn't change the compression ratio, which remains identical._
How I can achieve same compression ratio as on large file, still having individual 1MB blocks compressed into frames?
Contributor guide
Assessment
This issue has not been assessed yet.