ZSTD_compressSequencesAndLiterals() does not enforce its documented litCapacity >= litSize + 8 contract — the implemented check is 8 bytes weaker than the API documentation, its own error message, and the introducing commit
- Dominant language
- C
- Stars
- 27.9k
- Forks
- 2.6k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 8
Description
## **Describe the bug**
`ZSTD_compressSequencesAndLiterals()` does not enforce its own documented
`litBufCapacity >= litSize + 8` contract: the implemented check in
`lib/compress/zstd_compress.c:8128` only tests `litCapacity < litSize`,
i.e. it is **8 bytes weaker** than the contract stated in three independent
places:
1. **API documentation** — `lib/zstd.h` line 1700:
> `@litBufCapacity must be at least 8 bytes larger than @litSize.`
2. **The introducing commit** — `b7a9e69d` ("added parameter litCapacity
to ZSTD_compressSequencesAndLiterals() **to enforce the litCapacity >=
litSize+8 condition**.")
3. **The check's own error message** —
> "literals buffer is not large enough: **must be at least 8 bytes larger
> than litSize** (risk of read out-of-bound)"
```c
/* lib/compress/zstd_compress.c:8126-8130 */
if (litCapacity < litSize) { /* <-- should be: litCapacity < litSize + 8 */
RETURN_ERROR(workSpace_tooSmall,
"literals buffer is not large enough: "
"must be at least 8 bytes larger than litSize (risk of read out-of-bound)");
}
```
Calls that violate the documented contract by up to 8 bytes
(`litSize <= litCapacity < litSize + 8`) are silently accepted and produce
valid frames, instead of being rejected with the documented
`ZSTD_error_workSpace_tooSmall`.
## **To Reproduce**
Steps to reproduce the behavior:
1. Clone zstd and build the static library:
```sh
git clone https://github.com/facebook/zstd && cd zstd
make -C lib libzstd.a # or: make -C lib
```
2. Save the following as `reproducer.c` in the repo root:
```c
/* reproducer.c — demonstrates that ZSTD_compressSequencesAndLiterals()
* accepts litCapacity values that violate its documented
* "at least 8 bytes larger than litSize" contract. */
#define ZSTD_STATIC_LINKING_ONLY
#include "zstd.h"
#include
int main(void)
{
/* one explicit-delimiter block: 284 literals + a 16-byte match at
* offset 8, then 16 trailing literals carried by the delimiter */
ZSTD_Sequence seqs[2] = {
{ /*offset=*/8, /*litLength=*/284, /*matchLength=*/16, /*rep=*/0 },
{ /*offset=*/0, /*litLength=*/ 16, /*matchLength=*/ 0, /*rep=*/0 },
};
unsigned char literals[300 + 8]; /* +8 so the probe itself is memory-safe */
unsigned char dst[1024];
const size_t litSize = 300, srcSize = 316;
size_t i, r;
ZSTD_CCtx* cctx = ZSTD_createCCtx();
for (i = 0; i < litSize; i++) literals[i] = (unsigned char)('a' + (i % 26));
ZSTD_CCtx_setParameter(cctx, ZSTD_c_compressionLevel, 3);
ZSTD_CCtx_setParameter(cctx, ZSTD_c_blockDelimiters, ZSTD_sf_explicitBlockDelimiters);
/* documented minimum is litSize + 8; pass litSize + 7 on purpose */
r = ZSTD_compressSequencesAndLiterals(cctx, dst, sizeof(dst),
seqs, 2, literals, litSize,
litSize + 7, srcSize);
if (ZSTD_isError(r))
printf("litCapacity = litSize+7 : rejected (%s) -> contract enforced\n",
ZSTD_getErrorName(r));
else
printf("litCapacity = litSize+7 : ACCEPTED (%zu-byte frame) -> "
"documented contract NOT enforced\n", r);
ZSTD_freeCCtx(cctx);
return 0;
}
```
3. Compile and run:
```sh
gcc -O2 -Ilib reproducer.c lib/libzstd.a -o reproducer -lpthread
./reproducer
```
4. Scroll up on the log to the reproducer's output line:
```
litCapacity = litSize+7 : ACCEPTED (223-byte frame) -> documented contract NOT enforced
```
5. See error: there is **no error** — the contract-violating call is
accepted and produces a valid, round-tripping 223-byte frame. The same
happens with `litCapacity = litSize` (violating by the full 8 bytes).
With the one-line fix below, both calls are rejected with
`ZSTD_error_workSpace_tooSmall` while a compliant
`litCapacity = litSize + 8` call keeps working.
## **Expected behavior**
`ZSTD_compressSequencesAndLiterals()` should reject `litCapacity` values
below `litSize + 8` with `ZSTD_error_workSpace_tooSmall`, exactly as its
documentation (`lib/zstd.h:1700`), its introducing commit (`b7a9e69d`) and
its own error message all promise:
```diff
--- a/lib/compress/zstd_compress.c
+++ b/lib/compress/zstd_compress.c
@@ -8125,7 +8125,7 @@ ZSTD_compressSequencesAndLiterals(ZSTD_CCtx* cctx,
/* Transparent initialization stage, same as compressStream2() */
DEBUGLOG(4, "ZSTD_compressSequencesAndLiterals (dstCapacity=%zu)", dstCapacity);
assert(cctx != NULL);
- if (litCapacity < litSize) {
+ if (litCapacity < litSize + 8) {
RETURN_ERROR(workSpace_tooSmall, "literals buffer is not large enough: must be at least 8 bytes larger than litSize (risk of read out-of-bound)");
}
FORWARD_IF_ERROR(ZSTD_CCtx_init_compressStream2(cctx, ZSTD_e_end, decompressedSize), "CCtx initialization failed");
```
## **Screenshots and charts**
Not applicable — the reproducer output above shows the behavior
(`ACCEPTED ... contract NOT enforced` on current `dev`; after the one-line
fix the same call prints `rejected (workSpace buffer is not large enough)
-> contract enforced`).
## **Desktop (please complete the following information):**
- OS: Linux (x86-64)
- Version: Ubuntu 20.04
- Compiler: gcc 9.4.0
- Flags: `-O2` for the reproducer; the fuller PoC (see Additional context) was additionally run with `-g -O1 -fsanitize=address,undefined -fno-sanitize-recover=all`
- Other relevant hardware specs: n/a (no hardware dependency)
- Build system: `make -C lib` (static library `lib/libzstd.a`); library sources compiled directly in the PoC verification as well
## **Additional context**
* Affected version: current `dev` (verified byte-identical at
`lib/compress/zstd_compress.c:8128` on `dev` @ `d9c0c7e2`, 2026-09-09).
I could not find an existing issue or PR covering this mismatch (closest:
open PR #4741 fixes a different validation gap in the same function).
* Full verification PoC (attached): a three-phase test that (1) confirms a
compliant `litCapacity = litSize + 8` call succeeds, (2) demonstrates both
`litSize + 7` and `litSize` violations are accepted on the unpatched
build and rejected with `ZSTD_error_workSpace_tooSmall` after the
one-line fix, and (3) runs an ASan sweep of 360 configurations
(30 literal sizes x 3 data patterns x 4 levels, literals buffer of
exactly `litSize` bytes) probing for reads inside the documented-but-
unenforced margin `[litSize, litSize+8)`.
* **Honest impact statement:** the sweep produced **zero** ASan reports —
literal reads are currently bounded by `litSize` in the existing encoder
paths, so this is a contract-enforcement / defense-in-depth gap rather
than a demonstrated out-of-bounds read. The +8 slack exists because the
literal encoders historically used 8-byte-wide loads near the tail; the
check is the only guard between the caller and
`ZSTD_entropyCompressSeqStore_internal()`, and today it enforces none of
the slack that the documentation, the error message and commit
`b7a9e69d` all promise.
* Suggested title for the issue: "ZSTD_compressSequencesAndLiterals() does
not enforce the documented litBufCapacity >= litSize + 8 contract (check
is missing the + 8)".
Contributor guide
Research direction
Start at the litCapacity check in lib/compress/zstd_compress.c:8128 and compare it with the contract in lib/zstd.h. Build with make -C lib, run the reproducer described in the issue, and verify that capacities below litSize + 8 are rejected while litSize + 8 still succeeds.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 1/5
- Estimated time
- Under an hour
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 88/100