cloudflare / cloudflare/workerd
🐛 Bug Report — Runtime APIs: node:zlib zstd ignores the dictionary option (silent on compress, "Data corruption detected" on decompress)
- Dominant language
- C++
- Stars
- 8.7k
- Forks
- 739
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 174
Description
`node:zlib`'s zstd functions accept a `dictionary` option and drop it. On compression
that's silent, and on decompression it surfaces as a corruption error against bytes
that aren't corrupt.
Node grew this option in [nodejs/node@201304537e](https://github.com/nodejs/node/commit/201304537e)
and workerd's zstd bindings (#6007) predate it, so I assume this is a gap rather than a
regression.
### Versions
workerd `1.20260801.1`, via wrangler `4.120.0` / miniflare `5.20260801.1-alpha`.
`compatibility_date = "2026-08-01"`, `compatibility_flags = ["nodejs_compat"]`.
Compared against node `v26.7.0` running the identical program.
### Repro
```js
import { zstdCompressSync, zstdDecompressSync, constants } from "node:zlib";
// A frame produced by node with a dictionary, base64'd in so the worker can try to read it.
const FRAME = "";
export default {
fetch() {
const probe = Buffer.from("the quick brown fox jumps over the lazy dog ".repeat(200));
const wrong = Buffer.from("completely unrelated filler bytes ".repeat(200));
const lvl = { [constants.ZSTD_c_compressionLevel]: 19 };
const out = {};
out.compress = {
none: zstdCompressSync(probe, { params: lvl }).length,
good: zstdCompressSync(probe, { dictionary: probe, params: lvl }).length,
wrong: zstdCompressSync(probe, { dictionary: wrong, params: lvl }).length,
};
try {
const back = zstdDecompressSync(Buffer.from(FRAME, "base64"), { dictionary: probe });
out.decompress = { ok: true, bytes: back.length, matches: back.equals(probe) };
} catch (e) {
out.decompress = { ok: false, error: String(e?.message ?? e) };
}
return Response.json(out);
},
};
```
Generate `FRAME` under node:
```js
const { zstdCompressSync, constants } = require("node:zlib");
const probe = Buffer.from("the quick brown fox jumps over the lazy dog ".repeat(200));
zstdCompressSync(probe, { dictionary: probe, params: { [constants.ZSTD_c_compressionLevel]: 19 } })
.toString("base64"); // 19 bytes
```
### Results
| | node v26.7.0 | workerd 1.20260801.1 |
|---|---|---|
| compress, no dictionary | 63 | 63 |
| compress, **correct** dictionary | **19** | **63** |
| compress, wrong dictionary | 63 | 63 |
| decompress a dictionary-compressed frame | ok, 8800 bytes, byte-exact | throws |
workerd's decompress error:
```
Zstd decompression failed: Data corruption detected
```
### Why this isn't just a missing-feature note
**On compress, nothing tells you.** A frame compressed *without* a dictionary decodes
perfectly *with* one, so no consumer errors and no log line looks unusual. The only
signal is a byte count that never shrank. The only reliable detection I've found is a
feature-detect that compresses a buffer against itself and asserts the output collapses.
**On decompress, the error blames the wrong thing.** "Data corruption detected" points
at the payload. The payload is fine. Anyone debugging that will go looking at their
storage or their transport before they suspect that the option they passed was dropped.
The practical consequence is that a Worker can't participate in
[RFC 9842](https://www.rfc-editor.org/rfc/rfc9842.html) Compression Dictionary Transport
in either direction: it can't produce a `dcz` delta, and it can't read one. That's a
shame specifically for Workers, since computing a delta against a dictionary the client
already holds is an edge-shaped job.
### Suggested fix
1. Wire `ZSTD_CCtx_loadDictionary` / `ZSTD_DCtx_loadDictionary` (and the `_refPrefix`
variants, if you want raw-content dictionaries, which is what RFC 9842 uses) into the
bindings from #6007, matching node's implementation linked above.
2. If that's not near-term, please **throw** on a `dictionary` option the binding can't
honour, rather than accepting it. `ERR_INVALID_ARG_VALUE` at the call site is far
cheaper to debug than either symptom above.
I'm more than happy to take a stab via a PR for (2) if that's a useful stopgap.
### Related
- #6007 / #6117, the original ZSTD bindings
- #4013, implement zstd in node:zlib
- #6769 and #6773, a separate zstd "Unexpected end of file" bug, mentioned only so this
isn't triaged as a duplicate of it
- [oven-sh/bun#34427](https://github.com/oven-sh/bun/pull/34427), where bun had the
identical silent-ignore for both brotli and zstd and fixed it by loading the dictionary
into the context. Its test additions may be a useful reference.
Contributor guide
Research direction
Start with the zstd bindings from #6007 and the node:zlib zstd entry points, then run the provided compression and decompression repro against the Node comparison. Done means a dictionary changes compression and successfully decodes the supplied frame; if support is deferred, the dictionary option should fail explicitly instead of being ignored.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, node.js
- Domain
- api, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100