nodejs / nodejs/node

zstd decoding complete frame in read stream halts stream

Open
#64,741 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stream zlib
Dominant language
JavaScript
Stars
122k
Forks
37.3k
Avg merge
4d 2h
Merged PRs (30d)
283

Description

Version

v26.5.0

Platform
Darwin Mac.lan 24.6.0 Darwin Kernel Version 24.6.0: Tue Apr 21 20:19:12 PDT 2026; root:xnu-11417.140.69.710.16~1/RELEASE_ARM64_T6041 arm64
Subsystem

No response

What steps will reproduce the bug?
import { Buffer } from 'node:buffer';
import * as consumers from 'node:stream/consumers';
import * as zlib from 'node:zlib';

function consume(codec, writes) {
	const inflate = codec.createDecompress();
	const result = consumers.buffer(inflate);
	writes.forEach(write => inflate.write(write));
	inflate.end();
	return result;
}

function describe(buffer) {
	const aa = buffer.includes(0x41);
	const bb = buffer.includes(0x42);
	const which = function() {
		if (aa && bb) {
			return 'frameA + frameB';
		} else if (aa) {
			return 'frameA only';
		} else if (bb) {
			return 'frameB only';
		} else {
			return 'nothing';
		}
	}();
	return `${buffer.length} bytes (${which})`;
}

async function attempt(codec, writes) {
	try {
		return describe(await consume(codec, writes));
	} catch (error) {
		return `threw: ${error.message}`;
	}
}

const tests = [ {
	name: 'zstd',
	compress: buffer => zlib.zstdCompressSync(buffer),
	createDecompress: () => zlib.createZstdDecompress(),
}, {
	name: 'gzip',
	compress: buffer => zlib.gzipSync(buffer),
	createDecompress: () => zlib.createGunzip(),
} ];

console.log(`node ${process.version}`);
for (const codec of tests) {
	// 'A' / 'B'
	const frameA = codec.compress(Buffer.alloc(2000, 0x41));
	const frameB = codec.compress(Buffer.alloc(2000, 0x42));
	const concatenated = Buffer.concat([ frameA, frameB ]);

	// BUG: both frames in a single write — the boundary between them falls inside one input buffer.
	const bug = await attempt(codec, [ concatenated ]);
	// COUNTER-EXAMPLE: the same two frames, one per write — the boundary lands at a write boundary.
	const counter = await attempt(codec, [ frameA, frameB ]);

	console.log(`${codec.name}:`);
	console.log(`  stream, both frames in ONE write   -> ${bug}`);
	console.log(`  stream, one frame per write        -> ${counter}`);
}
How often does it reproduce? Is there a required condition?

This reproduces every time

What is the expected behavior? Why is that the expected behavior?

Decoding concatenated zstd payloads should decode correctly as one stream. Indeed, zstdcat will decode this correctly (omitted from example but you can take my word for it).

What do you see instead?
marcel[1:42:21PM] [~/xx] ~/Downloads/node-v26.5.0-darwin-arm64/bin/node a.mjs
node v26.5.0
zstd:
  stream, both frames in ONE write   -> 2000 bytes (frameA only)
  stream, one frame per write        -> 4000 bytes (frameA + frameB)
gzip:
  stream, both frames in ONE write   -> 4000 bytes (frameA + frameB)
  stream, one frame per write        -> 4000 bytes (frameA + frameB)
Additional information

This will cause spurious failures while decoding zstd streams. The odds of it happening are a function of stream read size vs zstd chunk size. In my case I ran into this while decoding a stream after about 3GB and tracked it down to this bug.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the reproduction script against the zlib streaming APIs, comparing concatenated zstd frames in one write with one frame per write. Trace the Node.js zstd decompression stream handling and add a regression test for the single-write case; done means both frames produce 4000 bytes, matching gzip and zstdcat behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.