facebook / facebook/zstd

Compression fails on linux/i386 with ZSTD_maxCLevel + ZSTD_c_nbWorkers

Open
#4,628 2 comments 1 reaction 0 assignees View on GitHub
Dominant language
C
Stars
27.9k
Forks
2.6k
Avg merge
1d 3h
Merged PRs (30d)
8

Description

**Describe the bug**

Compression fails on 32-bit architectures at level 22 with threading.
I've traced the problem to a `malloc(1<<31)` failure.

**To Reproduce**

On linux/i386, such as `docker run -it --platform=linux/i386 debian:testing`, create the following source file:

example.c (click to expand/collapse)

```c
#include
#include

int main(void)
{
size_t zret;

unsigned char const buf_in[] = "Sample Input Buffer Content";
ZSTD_inBuffer zin = { buf_in, sizeof(buf_in), 0 };
unsigned char buf_out[4096];
ZSTD_outBuffer zout = { buf_out, sizeof(buf_out), 0 };

ZSTD_CStream* zcs = ZSTD_createCStream();

/* Activate ZSTD_MULTITHREAD code paths. */
ZSTD_CCtx_setParameter(zcs, ZSTD_c_nbWorkers, 1);

/* Set maximum compression level (22). */
zret = ZSTD_initCStream(zcs, ZSTD_maxCLevel());
assert(!ZSTD_isError(zret) && "ZSTD_initCStream failed");

/* Try compressing. */
zret = ZSTD_compressStream(zcs, &zout, &zin);
assert(!ZSTD_isError(zret) && "ZSTD_compressStream failed");

ZSTD_freeCStream(zcs);

return 0;
}
```

Compile and link against a distro-packaged zstd 1.5.7. Run the binary:

```console
$ gcc -g example.c -o example -lzstd && ./example
example: example.c:24: main: Assertion `!ZSTD_isError(zret) && "ZSTD_compressStream failed"' failed.
```

Compression fails.

**Expected behavior**

Compression succeeds.

**Additional context**

The failure occurs on [this line](https://github.com/facebook/zstd/blob/v1.5.7/lib/compress/zstdmt_compress.c#L1331) with `capacity == 0x80000000`, `sectionsSize == 0x20000000`, `mtctx->targetSectionSize == 0x20000000`, `nbWorkers == 1`, `nbSlackBuffers == 3`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.