Cyan4973 / Cyan4973/FiniteStateEntropy
question: FSE doesn't always compress better than huf - expected?
- Dominant language
- C
- Stars
- 1.5k
- Forks
- 165
- PR merge metrics
- No merged PRs in 30d
Description
I can't share the data, but in short, it's 9108363519 bytes (~9GB) of almost-uncompressible data (IIRC it's the 9GB tail of a larger already-compressed stream).
% ./fse -e ./almost-uncompressable
Compressed 9108363519 bytes into 8992064047 bytes ==> 98.72%
% ./fse -h ./almost-uncompressable
Compressed 9108363519 bytes into 8943423537 bytes ==> 98.19%
% ./fse -z ./almost-uncompressable
Compressed 9108363519 bytes into 8944678105 bytes ==> 98.20%
Granted that I don't know the intimate details of FSE and this is a near-pathological case, but I'd have expected the two huffman implementations to fare rather worse than FSE on these almost-but-not-quite-uniform distributions of data.
Am I wrong?
Cheers.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.