facebook / facebook/zstd

Feature request: native multithreaded decompression for standard frames

Abierto
#4,775 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
27.9k
Forks
2.6k
Merge medio
1 d 3 h
PR fusionados (30 d)
8

Descripción

**Is your feature request related to a problem? Please describe.**

Some workloads compress a large archive once and decompress it many times. Zstandard supports multithreaded compression through `-T#`, but decompression remains single-threaded and can become a bottleneck on modern multicore systems with fast storage.

In one test with an approximately 4 GB archive, multithreaded zstd compression took about 21 seconds, while single-threaded zstd decompression plus tar extraction took about 26 seconds, compared with about 31 seconds using pigz. Compression completing faster than the extraction path is a useful signal that decompression is worth optimizing, although the extraction measurement also includes tar and filesystem writes.

This is a follow-up to #2470, where several users requested that multithreaded decompression be reconsidered after the issue was closed. Related discussions also exist in #2245 and #2499.

**Describe the solution you would like**

Add opt-in multithreaded decompression to the standard zstd CLI and API, ideally for streams produced by the normal zstd encoder. A CLI interface such as `zstd -d -T0` would be intuitive and consistent with compression.

**Describe alternatives you have considered**

`pzstd`, multiple independent frames, or manually splitting the input can enable parallel decompression, but these approaches require a different encoding strategy and may affect compression ratio or integration complexity.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.