facebook / facebook/zstd

Optimize ZSTD_compress2() with ZSTD_c_stableOutBuffer and ZSTD_c_stableInBuffer

Offen
#2,353 2 Kommentare 2 Reaktionen 1 zugewiesene Person Beansprucht von @terrelln Auf GitHub ansehen
enhancement feature request
Vorherrschende Sprache
C
Sterne
27.9k
Forks
2.6k
Ø Merge
1 T. 3 Std.
Gemergte PRs (30 T.)
8

Beschreibung

* [x] Add `ZSTD_c_stableOutBuffer`, which tells the compressor that the output buffer will never change.
* [x] Add `ZSTD_c_stableInBuffer`, which tells the compressor that the input buffer will never change. This is subtle. You may think that `ZSTD_e_end` is enough. But that isn't a strong enough guarantee when streaming can't complete in one pass (because the output buffer is too small for example). Passing `ZSTD_e_end` just guarantees that you pass the same *unconsumed* bytes, it doesn't say anything about the bytes already consumed.
* [x] Use `ZSTD_c_stableInBuffer` to avoid allocating a window buffer.
* [x] Use `ZSTD_c_stableOutBuffer` to compress directly to the output buffer, even when not guaranteed to be large enough.
* [ ] Thread these optimizations through to zstdmt.
* [x] Set both parameters in `ZSTD_compress2()` (and reset them to their original values after).

With these two parameters set, the streaming API should be exactly as efficient as the single-pass API. Then we could consider using the streaming API to implement all other functions.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.