cloudflare / cloudflare/workerd
🐛 Bug Report — Runtime APIs: node:zlib zstdCompressSync fails with Unexpected end of file on valid input
- Dominant language
- C++
- Stars
- 8.7k
- Forks
- 739
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 174
Description
## Summary
`node:zlib` zstd compression can fail with `Unexpected end of file` for valid, incompressible inputs.
This reproduces both on Workers production and with `wrangler dev --local`, so it looks like a `workerd` runtime bug rather than a production-only issue.
## Minimal reproduction
```js
import zlib from 'node:zlib';
function createRandomBuffer(size) {
const output = Buffer.alloc(size);
let state = 0x12345678;
for (let index = 0; index < size; index += 4) {
state ^= state << 13;
state ^= state >>> 17;
state ^= state << 5;
output[index] = state & 0xff;
if (index + 1 < size) output[index + 1] = (state >>> 8) & 0xff;
if (index + 2 < size) output[index + 2] = (state >>> 16) & 0xff;
if (index + 3 < size) output[index + 3] = (state >>> 24) & 0xff;
}
return output;
}
zlib.zstdCompressSync(createRandomBuffer(40944));
zlib.zstdCompressSync(createRandomBuffer(40960));
```
## Observed behavior
- `zlib.zstdCompressSync(createRandomBuffer(40944))` succeeds
- `zlib.zstdCompressSync(createRandomBuffer(40960))` throws `Unexpected end of file`
- `zlib.zstdCompress(createRandomBuffer(40960), cb)` also fails with the same error
- `zlib.gzipSync()` succeeds for the same inputs
I also reproduced the same class of failure with an R2-backed input in Workers, but the generated-buffer case above is the smallest reproducer I found.
## Expected behavior
Valid input should compress successfully, or return an actual zstd compression error if `ZSTD_isError()` reports one.
It should not be converted into `Unexpected end of file` while compressing valid input.
## Suspected cause
In [workerd/src/workerd/api/node/zlib-util.c++](workerd/src/workerd/api/node/zlib-util.c++), `ZstdEncoderContext::getError()` treats any non-zero `lastResult` at `ZSTD_e_end` as EOF:
```c++
if (flush_ == ZSTD_e_end && lastResult != 0) {
return CompressionError("Unexpected end of file"_kj, "Z_BUF_ERROR"_kj, Z_BUF_ERROR);
}
```
But `lastResult != 0` from `ZSTD_compressStream2()` means there is still pending output to flush, not truncated input.
The sync loop in the same file only continues while `ctx.getAvailOut() == 0`:
```c++
do {
...
ctx.work();
...
} while (ctx.getAvailOut() == 0);
```
So it looks possible for the loop to stop while zstd still has remaining output to emit, and then `getError()` turns that state into `Unexpected end of file`.
## Source references
- [workerd/src/workerd/api/node/zlib-util.c++](workerd/src/workerd/api/node/zlib-util.c++)
- [workerd/src/workerd/api/node/tests/zlib-zstd-nodejs-test.js](workerd/src/workerd/api/node/tests/zlib-zstd-nodejs-test.js)
## Additional note
`zstdSync()` validates `options.chunkSize`, but constructs `GrowableBuffer` with `ZLIB_PERFORMANT_CHUNK_SIZE` instead of the provided `chunkSize`, so changing `chunkSize` does not seem to affect this path.
Contributor guide
Assessment
This issue has not been assessed yet.