facebook / facebook/zstd

ZSTD_insertBtAndGetAllMatches assertion failed

Open
#4,610 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
27.9k
Forks
2.6k
Avg merge
1d 3h
Merged PRs (30d)
8

Description

**Describe the bug**
I have a huge file (1300MB) and zstd hits assertion failure randomly when compress that file.

**To Reproduce**
Steps to reproduce the behavior:
1. Downloads data https://kcwu.csie.org/~kcwu/tmp/zstd-crash.o5m.zst and decompress it. The sha256sum of the o5m file is `90746b4adaf1492cb73f4aae80917f32a98be41cb56e09e3b4f5d719d0494726`.
2. Run 'zstd -19 -T0 -f -k -q zstd-crash.o5m' repeatedly.
3. See error
```
zstd: ../lib/compress/zstd_opt.c:722: ZSTD_insertBtAndGetAllMatches: Assertion `memcmp(match, ip, matchLength) == 0' failed.
Caught SIGABRT signal, printing stack:
/lib/x86_64-linux-gnu/libc.so.6(pthread_kill+0x11c) [0x7e840202db2c]
/lib/x86_64-linux-gnu/libc.so.6(gsignal+0x1e) [0x7e8401fd427e]
/lib/x86_64-linux-gnu/libc.so.6(abort+0xdf) [0x7e8401fb78ff]
/lib/x86_64-linux-gnu/libc.so.6(+0x2881b) [0x7e8401fb781b]
/lib/x86_64-linux-gnu/libc.so.6(+0x3b517) [0x7e8401fca517]
zstd(+0x54e7b) [0x5f6d40700e7b]
zstd(+0x5f839) [0x5f6d4070b839]
zstd(+0x182a1) [0x5f6d406c42a1]
zstd(+0x1aae6) [0x5f6d406c6ae6]
zstd(+0x1bda9) [0x5f6d406c7da9]
zstd(+0x6239d) [0x5f6d4070e39d]
zstd(+0x9eb8) [0x5f6d406b5eb8]
/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x7e840202baa4]
/lib/x86_64-linux-gnu/libc.so.6(+0x129c6c) [0x7e84020b8c6c]
```

I tested the same command repeatedly (with cool down between runs). zstd crashed 3 out of 100 times.

**Expected behavior**
zstd should not crash.

**Desktop (please complete the following information):**
- OS: ubuntu-26.04
- Other relevant hardware specs: AMD Ryzen 7 3700X 8-Core Processor. total memory 128GB. free memory is 36GB when zstd runs.
- zstd is installed from ubuntu package. version: `*** Zstandard CLI (64-bit) v1.5.5, by Yann Collet ***`

**Additional context**
- the file is a static file. No other processes touch the file when zstd compress.
- I stress tested the system and it is stable (memory is good). zstd won't crash with other similar files (repeated run two days).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.