Non-deterministic output with ZSTD_createCDict_byReference
- Dominant language
- C
- Stars
- 27.9k
- Forks
- 2.6k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 8
Description
**Describe the bug**
It's not a logical bug. Just the `ZSTD_createCDict_byReference` would output differently if the input data is directly adjacent to the dict. Making the output non-deterministic if the input is randomly allocated.
I found there's a opt-in param ZSTD_c_deterministicRefPrefix , that solves the problem for refPrefix. They share exactly same mechanism. But it doesn't apply to the CDict.
And the comment https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L2275 said
```
* If you really care about determinism when using a dictionary or prefix,
* like when doing delta compression, you should select this option.
```
So I believe it might also need to apply to the new API `ZSTD_createCDict_byReference`
**To Reproduce**
Compressing one 256 KB random buffer with a random 64 KB `ZSTD_createCDict_byReference()` dictionary,
contiguous vs separated:
```
level 2 | byReference | MISMATCH
level 3 | byReference | MISMATCH
level 4 | byReference | MISMATCH
level 6 | byReference | MISMATCH
level 2 | byReference, deterministicRefPrefix=1 | MISMATCH
level 2 | byCopy (control), default | IDENTICAL
```
**Expected behavior**
Opt-in deterministic mode
**Screenshots and charts**
If applicable, add screenshots and charts to help explain your problem.
**Desktop (please complete the following information):**
- OS: [Ubuntu]
- Version [24.04]
- Compiler [gcc13.3.0]
- Flags [any flag]
- Other relevant hardware specs [AVX2 , 16 cores]
- Build system [Any]
**Additional context**
I'll create a PR with unit test.
Contributor guide
Assessment
This issue has not been assessed yet.