facebook / facebook/zstd

Feature request: native multithreaded decompression for standard frames

Open
#4,775 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
27.9k
Forks
2.6k
Avg merge
1d 3h
Merged PRs (30d)
8

Description

**Is your feature request related to a problem? Please describe.**

Some workloads compress a large archive once and decompress it many times. Zstandard supports multithreaded compression through `-T#`, but decompression remains single-threaded and can become a bottleneck on modern multicore systems with fast storage.

In one test with an approximately 4 GB archive, multithreaded zstd compression took about 21 seconds, while single-threaded zstd decompression plus tar extraction took about 26 seconds, compared with about 31 seconds using pigz. Compression completing faster than the extraction path is a useful signal that decompression is worth optimizing, although the extraction measurement also includes tar and filesystem writes.

This is a follow-up to #2470, where several users requested that multithreaded decompression be reconsidered after the issue was closed. Related discussions also exist in #2245 and #2499.

**Describe the solution you would like**

Add opt-in multithreaded decompression to the standard zstd CLI and API, ideally for streams produced by the normal zstd encoder. A CLI interface such as `zstd -d -T0` would be intuitive and consistent with compression.

**Describe alternatives you have considered**

`pzstd`, multiple independent frames, or manually splitting the input can enable parallel decompression, but these approaches require a different encoding strategy and may affect compression ratio or integration complexity.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.