facebook / facebook/zstd

Non-deterministic output with ZSTD_createCDict_byReference

Offen
#4,738 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
C
Sterne
27.9k
Forks
2.6k
Ø Merge
1 T. 3 Std.
Gemergte PRs (30 T.)
8

Beschreibung

**Describe the bug**

It's not a logical bug. Just the `ZSTD_createCDict_byReference` would output differently if the input data is directly adjacent to the dict. Making the output non-deterministic if the input is randomly allocated.

I found there's a opt-in param ZSTD_c_deterministicRefPrefix , that solves the problem for refPrefix. They share exactly same mechanism. But it doesn't apply to the CDict.

And the comment https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L2275 said

```
* If you really care about determinism when using a dictionary or prefix,
* like when doing delta compression, you should select this option.
```
So I believe it might also need to apply to the new API `ZSTD_createCDict_byReference`

**To Reproduce**

Compressing one 256 KB random buffer with a random 64 KB `ZSTD_createCDict_byReference()` dictionary,
contiguous vs separated:

```
level 2 | byReference | MISMATCH
level 3 | byReference | MISMATCH
level 4 | byReference | MISMATCH
level 6 | byReference | MISMATCH
level 2 | byReference, deterministicRefPrefix=1 | MISMATCH
level 2 | byCopy (control), default | IDENTICAL
```

**Expected behavior**
Opt-in deterministic mode

**Screenshots and charts**
If applicable, add screenshots and charts to help explain your problem.

**Desktop (please complete the following information):**
- OS: [Ubuntu]
- Version [24.04]
- Compiler [gcc13.3.0]
- Flags [any flag]
- Other relevant hardware specs [AVX2 , 16 cores]
- Build system [Any]

**Additional context**

I'll create a PR with unit test.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.