facebook / facebook/zstd

Question: would a shape-aware preprocessor be a useful contrib/ tool?

Abierto
#4,659 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
C
Estrellas
27.9k
Forks
2.6k
Merge medio
1 d 3 h
PR fusionados (30 d)
8

Descripción

````markdown
## Question: would a shape-aware preprocessor be useful as a `contrib/` tool?

Hi zstd team,

I wanted to ask whether this kind of contribution would be of interest before opening a PR.

I have built a small open-source preprocessor library called **STRATA**:

https://github.com/rjamesy/strata

It is MIT-licensed and exposes a few reversible, shape-aware transforms:

- 2D predictors
- YCoCg-R colour decorrelation
- full octahedral cube rotation
- radial reordering for 3D voxels
- an auto-selector that keeps raw input as a candidate, so the selected output is never worse than the underlying codec alone

The result that prompted the question is that, for some shape-aware data, `STRATA-preprocess + zstd -1` beats `raw input + zstd -22` on both speed and ratio.

Examples:

```text
27 MB RGB photo:
YCoCg-R + zstd -1
= 463x faster, 5.4% smaller than raw zstd -22

Smooth 2D heightmap:
2D predictor + zstd -1
= 4.2x faster, 44.5% smaller than raw zstd -22

64^3 byte volume:
cube rotation + radial reordering + zstd -22
= 14.4% smaller than raw zstd -22, at roughly the same speed
````

The mechanism is not zstd-specific. The transforms expose structure that is hard for a byte-stream compressor to infer from raw input, especially when the redundancy is smoothness rather than exact repeated byte sequences.

The repo includes a reproduction script:

```bash
python3 bench/preprocess_demo.py
```

It writes a CSV covering the tested codec/dataset combinations.

My questions:

1. Would this kind of shape-aware preprocessor be a plausible fit for `contrib/`?

I am not proposing changes to zstd core. The transforms are separate reversible passes, roughly 30–300 LOC each, and the tool would emit zstd-compressible bytes.

2. Is the design preference for `contrib/` to remain fully byte-stream-general, or are optional shape-aware tools acceptable there?

I saw `contrib/seekable_format` and `contrib/pzstd`, so I wanted to ask rather than assume the boundary.

3. Are there specific data classes the team would want benchmarked before considering something like this?

My current corpus is small: terrain, CT-style volumes, RGB photos, and sensor CSV.

I am also happy to keep this as an external library if that is the better fit. I mainly wanted to check before opening a PR.

Cheers,
Richard

```
```

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.