facebook / facebook/zstd

Feature request: native multithreaded decompression for standard frames

Aperta
#4,775 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
C
Stelle
27.9k
Fork
2.6k
Merge medio
1g 3h
PR unite (30g)
8

Descrizione

**Is your feature request related to a problem? Please describe.**

Some workloads compress a large archive once and decompress it many times. Zstandard supports multithreaded compression through `-T#`, but decompression remains single-threaded and can become a bottleneck on modern multicore systems with fast storage.

In one test with an approximately 4 GB archive, multithreaded zstd compression took about 21 seconds, while single-threaded zstd decompression plus tar extraction took about 26 seconds, compared with about 31 seconds using pigz. Compression completing faster than the extraction path is a useful signal that decompression is worth optimizing, although the extraction measurement also includes tar and filesystem writes.

This is a follow-up to #2470, where several users requested that multithreaded decompression be reconsidered after the issue was closed. Related discussions also exist in #2245 and #2499.

**Describe the solution you would like**

Add opt-in multithreaded decompression to the standard zstd CLI and API, ideally for streams produced by the normal zstd encoder. A CLI interface such as `zstd -d -T0` would be intuitive and consistent with compression.

**Describe alternatives you have considered**

`pzstd`, multiple independent frames, or manually splitting the input can enable parallel decompression, but these approaches require a different encoding strategy and may affect compression ratio or integration complexity.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.