Cyan4973 / Cyan4973/FiniteStateEntropy

FSE_compress returns 0 even when maxDstSize is still equal to or slightly larger than final compression size

Open
#90 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
1.5k
Forks
165
PR merge metrics
No merged PRs in 30d

Description

While adding more unit tests, I found a corner case when trying to compress a document with a `maxDstSize` which is exactly the expected compressed size (found by a previous compression attempt): in this test, `FSE_compress(..)` returns 0 (uncompressible data) while previously it was able to compress the same input (given a larger destination buffer).

I'm doing a two-step process:
1) Allocate a buffer large enough (using `FSE_compressBound`), and compress the source by calling `FSE_compress(..., maxDstSize: FSE_compressBound(ORIGINAL_SIZE))`, and measure its final compressed size `COMPRESSED_SIZE`.
2) Repeat the process on the same buffer and payload, but this time calling `FSE_compress(..., maxDstSize: COMPRESSED_SIZE)`. I get a result code of __0__ which means that data is uncompressible.

I tried probing the minimum size that will allow compressing the buffer (which is known to be compressible), and each time I need to call `FSE_compress(..)` with at least `COMPRESSED_SIZE + 8`. At first I thought it could be a pointer alignment issue, but it is always + 8 bytes whatever compressed size is (by changing the source a bit).

In my test, raw original size is __4,288__ bytes, `FSE_compressBound` return __4,833__ bytes, and the compressed size is __2,821__ bytes. I need to pass at least 2,823+8 = __2,829__ bytes for `FSE_compress` to succeed (return value > 0).

Is this expected behavior? I'm not sure if the "+8 rule" is true, or if this is random chance with the inputs I'm passing in.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.