intel / intel/isa-l

Add support for inflate flush points

Open
#254 0 comments 1 reaction 0 assignees View on GitHub
enhancement
Dominant language
C
Stars
1.1k
Forks
346
PR merge metrics
No merged PRs in 30d

Description

Hi,

The gzip/deflate decompression speed of this library is amazing. That's why I'm trying to get it to work in as many places as possible in my multi-threaded gzip decompression tool [rapidgzip](https://github.com/mxmlnkn/rapidgzip).

However, in order to really use it in as many places as possible (without modifying the Assembler routines), I would require flush points to be implemented because I need those to decide suitable points to stop decompression at inside one of the decompression threads.

Zlib has this description for inflate:

> The flush parameter of inflate() can be Z_NO_FLUSH, Z_SYNC_FLUSH, Z_FINISH,
Z_BLOCK, or Z_TREES. Z_SYNC_FLUSH requests that inflate() flush as much
output as possible to the output buffer. Z_BLOCK requests that inflate()
stop if and when it gets to the next deflate block boundary. When decoding
the zlib or gzip format, this will cause inflate() to return immediately
after the header and before the first block. When doing a raw inflate,
inflate() will go ahead and process the first block, and will return when it
gets to the end of that block, or when it runs out of data.

I would need support for `Z_BLOCK` and `Z_TREES`. As a proof of concept, I tried to [implement this in my fork](https://github.com/mxmlnkn/isa-l/commit/9e1dea698ddcd608a245ebd029d885cb0231037e). However, it would be nicer to have this implemented in the upstream and by someone who knows the code inside out. E.g., I'm unsure about when exactly those `tmp_in_buffer` and `tmp_in_size` state members hold valid values. This is important to me because I not only need ISA-l to stop at those flush points, I also need to be able to infer the exact bit position in the encoded stream and byte position in the decoded stream from `read_in_length` and `avail_in` after it has stopped decompression at those flush points.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.