facebook / facebook/zstd

Optimize ZSTD_compress2() with ZSTD_c_stableOutBuffer and ZSTD_c_stableInBuffer

Open
#2,353 2 comments 2 reactions 1 assignee Claimed by @terrelln View on GitHub
enhancement feature request
Dominant language
C
Stars
27.9k
Forks
2.6k
Avg merge
1d 3h
Merged PRs (30d)
8

Description

* [x] Add `ZSTD_c_stableOutBuffer`, which tells the compressor that the output buffer will never change.
* [x] Add `ZSTD_c_stableInBuffer`, which tells the compressor that the input buffer will never change. This is subtle. You may think that `ZSTD_e_end` is enough. But that isn't a strong enough guarantee when streaming can't complete in one pass (because the output buffer is too small for example). Passing `ZSTD_e_end` just guarantees that you pass the same *unconsumed* bytes, it doesn't say anything about the bytes already consumed.
* [x] Use `ZSTD_c_stableInBuffer` to avoid allocating a window buffer.
* [x] Use `ZSTD_c_stableOutBuffer` to compress directly to the output buffer, even when not guaranteed to be large enough.
* [ ] Thread these optimizations through to zstdmt.
* [x] Set both parameters in `ZSTD_compress2()` (and reset them to their original values after).

With these two parameters set, the streaming API should be exactly as efficient as the single-pass API. Then we could consider using the streaming API to implement all other functions.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.