facebook / facebook/zstd

Non-deterministic output with ZSTD_createCDict_byReference

Aperta
#4,738 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
C
Stelle
27.9k
Fork
2.6k
Merge medio
1g 3h
PR unite (30g)
8

Descrizione

**Describe the bug**

It's not a logical bug. Just the `ZSTD_createCDict_byReference` would output differently if the input data is directly adjacent to the dict. Making the output non-deterministic if the input is randomly allocated.

I found there's a opt-in param ZSTD_c_deterministicRefPrefix , that solves the problem for refPrefix. They share exactly same mechanism. But it doesn't apply to the CDict.

And the comment https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L2275 said

```
* If you really care about determinism when using a dictionary or prefix,
* like when doing delta compression, you should select this option.
```
So I believe it might also need to apply to the new API `ZSTD_createCDict_byReference`

**To Reproduce**

Compressing one 256 KB random buffer with a random 64 KB `ZSTD_createCDict_byReference()` dictionary,
contiguous vs separated:

```
level 2 | byReference | MISMATCH
level 3 | byReference | MISMATCH
level 4 | byReference | MISMATCH
level 6 | byReference | MISMATCH
level 2 | byReference, deterministicRefPrefix=1 | MISMATCH
level 2 | byCopy (control), default | IDENTICAL
```

**Expected behavior**
Opt-in deterministic mode

**Screenshots and charts**
If applicable, add screenshots and charts to help explain your problem.

**Desktop (please complete the following information):**
- OS: [Ubuntu]
- Version [24.04]
- Compiler [gcc13.3.0]
- Flags [any flag]
- Other relevant hardware specs [AVX2 , 16 cores]
- Build system [Any]

**Additional context**

I'll create a PR with unit test.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.