cloudflare / cloudflare/workerd

🐛 Bug Report — Runtime APIs: node:zlib zstdCompressSync fails with Unexpected end of file on valid input

Open
#6,769 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
8.7k
Forks
739
Avg merge
2d 20h
Merged PRs (30d)
174

Description

## Summary

`node:zlib` zstd compression can fail with `Unexpected end of file` for valid, incompressible inputs.

This reproduces both on Workers production and with `wrangler dev --local`, so it looks like a `workerd` runtime bug rather than a production-only issue.

## Minimal reproduction

```js
import zlib from 'node:zlib';

function createRandomBuffer(size) {
const output = Buffer.alloc(size);
let state = 0x12345678;
for (let index = 0; index < size; index += 4) {
state ^= state << 13;
state ^= state >>> 17;
state ^= state << 5;
output[index] = state & 0xff;
if (index + 1 < size) output[index + 1] = (state >>> 8) & 0xff;
if (index + 2 < size) output[index + 2] = (state >>> 16) & 0xff;
if (index + 3 < size) output[index + 3] = (state >>> 24) & 0xff;
}
return output;
}

zlib.zstdCompressSync(createRandomBuffer(40944));
zlib.zstdCompressSync(createRandomBuffer(40960));
```

## Observed behavior

- `zlib.zstdCompressSync(createRandomBuffer(40944))` succeeds
- `zlib.zstdCompressSync(createRandomBuffer(40960))` throws `Unexpected end of file`
- `zlib.zstdCompress(createRandomBuffer(40960), cb)` also fails with the same error
- `zlib.gzipSync()` succeeds for the same inputs

I also reproduced the same class of failure with an R2-backed input in Workers, but the generated-buffer case above is the smallest reproducer I found.

## Expected behavior

Valid input should compress successfully, or return an actual zstd compression error if `ZSTD_isError()` reports one.

It should not be converted into `Unexpected end of file` while compressing valid input.

## Suspected cause

In [workerd/src/workerd/api/node/zlib-util.c++](workerd/src/workerd/api/node/zlib-util.c++), `ZstdEncoderContext::getError()` treats any non-zero `lastResult` at `ZSTD_e_end` as EOF:

```c++
if (flush_ == ZSTD_e_end && lastResult != 0) {
return CompressionError("Unexpected end of file"_kj, "Z_BUF_ERROR"_kj, Z_BUF_ERROR);
}
```

But `lastResult != 0` from `ZSTD_compressStream2()` means there is still pending output to flush, not truncated input.

The sync loop in the same file only continues while `ctx.getAvailOut() == 0`:

```c++
do {
...
ctx.work();
...
} while (ctx.getAvailOut() == 0);
```

So it looks possible for the loop to stop while zstd still has remaining output to emit, and then `getError()` turns that state into `Unexpected end of file`.

## Source references

- [workerd/src/workerd/api/node/zlib-util.c++](workerd/src/workerd/api/node/zlib-util.c++)
- [workerd/src/workerd/api/node/tests/zlib-zstd-nodejs-test.js](workerd/src/workerd/api/node/tests/zlib-zstd-nodejs-test.js)

## Additional note

`zstdSync()` validates `options.chunkSize`, but constructs `GrowableBuffer` with `ZLIB_PERFORMANT_CHUNK_SIZE` instead of the provided `chunkSize`, so changing `chunkSize` does not seem to affect this path.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.