facebook / facebook/zstd

Feature request: native multithreaded decompression for standard frames

オープン
#4,775 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
C
スター
27.9k
フォーク
2.6k
平均マージ
1日 3時間
マージ済み PR(30日)
8

説明

**Is your feature request related to a problem? Please describe.**

Some workloads compress a large archive once and decompress it many times. Zstandard supports multithreaded compression through `-T#`, but decompression remains single-threaded and can become a bottleneck on modern multicore systems with fast storage.

In one test with an approximately 4 GB archive, multithreaded zstd compression took about 21 seconds, while single-threaded zstd decompression plus tar extraction took about 26 seconds, compared with about 31 seconds using pigz. Compression completing faster than the extraction path is a useful signal that decompression is worth optimizing, although the extraction measurement also includes tar and filesystem writes.

This is a follow-up to #2470, where several users requested that multithreaded decompression be reconsidered after the issue was closed. Related discussions also exist in #2245 and #2499.

**Describe the solution you would like**

Add opt-in multithreaded decompression to the standard zstd CLI and API, ideally for streams produced by the normal zstd encoder. A CLI interface such as `zstd -d -T0` would be intuitive and consistent with compression.

**Describe alternatives you have considered**

`pzstd`, multiple independent frames, or manually splitting the input can enable parallel decompression, but these approaches require a different encoding strategy and may affect compression ratio or integration complexity.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。