facebook / facebook/zstd

Non-deterministic output with ZSTD_createCDict_byReference

オープン
#4,738 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
C
スター
27.9k
フォーク
2.6k
平均マージ
1日 3時間
マージ済み PR(30日)
8

説明

**Describe the bug**

It's not a logical bug. Just the `ZSTD_createCDict_byReference` would output differently if the input data is directly adjacent to the dict. Making the output non-deterministic if the input is randomly allocated.

I found there's a opt-in param ZSTD_c_deterministicRefPrefix , that solves the problem for refPrefix. They share exactly same mechanism. But it doesn't apply to the CDict.

And the comment https://github.com/facebook/zstd/blob/dev/lib/zstd.h#L2275 said

```
* If you really care about determinism when using a dictionary or prefix,
* like when doing delta compression, you should select this option.
```
So I believe it might also need to apply to the new API `ZSTD_createCDict_byReference`

**To Reproduce**

Compressing one 256 KB random buffer with a random 64 KB `ZSTD_createCDict_byReference()` dictionary,
contiguous vs separated:

```
level 2 | byReference | MISMATCH
level 3 | byReference | MISMATCH
level 4 | byReference | MISMATCH
level 6 | byReference | MISMATCH
level 2 | byReference, deterministicRefPrefix=1 | MISMATCH
level 2 | byCopy (control), default | IDENTICAL
```

**Expected behavior**
Opt-in deterministic mode

**Screenshots and charts**
If applicable, add screenshots and charts to help explain your problem.

**Desktop (please complete the following information):**
- OS: [Ubuntu]
- Version [24.04]
- Compiler [gcc13.3.0]
- Flags [any flag]
- Other relevant hardware specs [AVX2 , 16 cores]
- Build system [Any]

**Additional context**

I'll create a PR with unit test.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。