facebook / facebook/zstd

Container format that facilitates dictionaries for multiple files

Abierto
#4,594 0 comentarios 1 reacción 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
27.9k
Forks
2.6k
Merge medio
1 d 3 h
PR fusionados (30 d)
8

Descripción

A major use case for compression is to create single-file archives (containers) which contain multiple files and support adding/removing individual files.

[RFC 9842](https://www.rfc-editor.org/rfc/rfc9842) adds dictionary support using HTTP; Jeremy Wagner of Google published [a blog post on Mar 6 2024 on shared dictionary compression](https://developer.chrome.com/blog/shared-dictionary-compression). This suggests similar advantages in archive files containing multiple .zstd-compressed items which share one or more common dictionaries. (Imagine a compressed source tree containing Java / Python / Javascript / HTML / CSS / Markdown / whatever.)

What would it take to create such a container format with support for compression dictionaries shared among the content items?

Many existing compression archives are .zip files and it is refreshing to see that the PKWARE format supports zstd (.zip method 93) as of around 2020. The main ecosystem pieces are lagging behind, however, in .zip support utilities:

- WinZip -- ahead of curve with [zstd in zipx in WinZip 24](https://kb.winzip.com/130539) (Sep 2019)
- libarchive -- zstd in [libarchive 3.6.2](https://github.com/libarchive/libarchive/releases/tag/v3.6.2) (Dec 2022), used in Windows 11 as of [KB5031455 in Oct 2023](https://support.microsoft.com/en-us/topic/october-31-2023-kb5031455-os-builds-22621-2506-and-22631-2506-preview-6513c5ec-c5a2-4aaf-97f5-44c13d29e0d4) (see "File Explorer" in release notes)
- Python zipfile library -- [zstd in Python 3.14](https://docs.python.org/3/library/zipfile.html#zipfile.ZIP_ZSTANDARD) (Oct 2025)
- 7zip -- zstd supported in [7zip 24.05](https://github.com/ip7z/7zip/releases/tag/24.05) (May 2024) for unpacking from existing .zip files, not yet for compressing new content
- info-zip -- zstd not supported yet (?), but fork does https://github.com/csabahruska/unzip-zstd

Is there a discussion list for zstd that could be used to collaborate with the major ecosystem providers and discuss potential proposals?

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.