facebook / facebook/zstd

How to achieve better compression on 1MB block by re-using compression context

Open
#3,140 2 comments 0 reactions 1 assignee Claimed by @Cyan4973 View on GitHub
question
Dominant language
C
Stars
27.9k
Forks
2.6k
Avg merge
1d 3h
Merged PRs (30d)
8

Description

I am compressing 1G JSON file, with zstd command, and getting 27.55 MB file (2.75% compression ratio - very impressive!!!!).

When I am compressing same file, breaking input into 1MB blocks, and compressing each block individually into frame, I am getting 41.7MB file, around 50% worse compression ratio than zstd command line tool.

Increasing block size to 10MB make compression ratio go closer to original but still far.

I suspect zstd utility is building dictionary on very large data sample, much larger than 10MB, and it makes much better compression ratio.

Obviously you need to reuse compression context, in hope that dictionary will be reused between invocations of ZSTD_compress.

Code (v1.5.2) explicitly saying it is not going to work:

_Note : re-using context is just a speed / resource optimization.
It doesn't change the compression ratio, which remains identical._

How I can achieve same compression ratio as on large file, still having individual 1MB blocks compressed into frames?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.