Add Tools to Scale Compression Resources on Constrained Systems
- Lenguaje dominante
- C
- Estrellas
- 27.9k
- Forks
- 2.6k
- Merge medio
- 1 d 3 h
- PR fusionados (30 d)
- 8
Descripción
**Is your feature request related to a problem? Please describe.**
Developers who want to use zstd in its higher compression modes face difficult choices when their application is shipped to and runs on highly disparate devices. On high powered server-class hardware, level 19 (or even higher) may be appropriate, but the same level on a constrained device (e.g., a Raspberry Pi) may take unconscionably long to complete--or even crash!
**Describe the solution you'd like**
We should offer users of zstd tools to scale down the resource consumption of compression based on the constraints of the system. The three relevant constraints are probably time, memory, and threads. The most pressing is probably memory consumption, since it can lead to crashes.
There are a number of ways to indicate these constraints and to reconcile them with the overall compression intent. My expectation is that the most straightforward is for users to select the compression level etc. that they'd like to use in the absence of constraints and then supply any constraints. It would then be zstd's job to reconcile them if they conflict. E.g.:
```c
ZSTD_compressionParameters ZSTD_selectCParamsMatchingConstraints(int cLevel, size_t inputSize, size_t maxMem) {
ZSTD_compressionParameters cParams;
do {
/* cctx doesn't get any smaller after -1 */
cParams = ZSTD_getCParams(cLevel, inputSize, 0);
size_t size = ZSTD_estimateCStreamSize_usingCParams(cParams);
if (size <= maxMem) {
break;
}
} while (--cLevel >= -1);
return cParams;
}
```
A follow-up topic would be whether zstd could determine these constraints itself (like `-T0` does for threads).
**Describe alternatives you've considered**
Users have the tools to make these choices themselves. However, it would put significant burden on them to have to understand how compression parameters map to memory usage etc.
**Additional context**
This came up on the [ubuntu mailing list](https://lists.ubuntu.com/archives/ubuntu-devel/2021-December/041726.html) in the context of rebuilding the initramfs.
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.