facebook / facebook/zstd

Add Tools to Scale Compression Resources on Constrained Systems

オープン
#2,933 コメント 3 件 リアクション 1 件 担当者 1 名 @felixhandte が担当を希望しています GitHub で見る
feature request
主要言語
C
スター
27.9k
フォーク
2.6k
平均マージ
1日 3時間
マージ済み PR(30日)
8

説明

**Is your feature request related to a problem? Please describe.**
Developers who want to use zstd in its higher compression modes face difficult choices when their application is shipped to and runs on highly disparate devices. On high powered server-class hardware, level 19 (or even higher) may be appropriate, but the same level on a constrained device (e.g., a Raspberry Pi) may take unconscionably long to complete--or even crash!

**Describe the solution you'd like**
We should offer users of zstd tools to scale down the resource consumption of compression based on the constraints of the system. The three relevant constraints are probably time, memory, and threads. The most pressing is probably memory consumption, since it can lead to crashes.

There are a number of ways to indicate these constraints and to reconcile them with the overall compression intent. My expectation is that the most straightforward is for users to select the compression level etc. that they'd like to use in the absence of constraints and then supply any constraints. It would then be zstd's job to reconcile them if they conflict. E.g.:

```c
ZSTD_compressionParameters ZSTD_selectCParamsMatchingConstraints(int cLevel, size_t inputSize, size_t maxMem) {
ZSTD_compressionParameters cParams;
do {
/* cctx doesn't get any smaller after -1 */
cParams = ZSTD_getCParams(cLevel, inputSize, 0);
size_t size = ZSTD_estimateCStreamSize_usingCParams(cParams);
if (size <= maxMem) {
break;
}
} while (--cLevel >= -1);
return cParams;
}
```

A follow-up topic would be whether zstd could determine these constraints itself (like `-T0` does for threads).

**Describe alternatives you've considered**
Users have the tools to make these choices themselves. However, it would put significant burden on them to have to understand how compression parameters map to memory usage etc.

**Additional context**
This came up on the [ubuntu mailing list](https://lists.ubuntu.com/archives/ubuntu-devel/2021-December/041726.html) in the context of rebuilding the initramfs.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。