zlib: createZstdCompress appends an empty frame when end() is called with writes still queued

Version

v26.9.0 (also v24.15.0)

Platform

Darwin 25.6.0 arm64 (also seen on Linux x64)

Subsystem

zlib

What steps will reproduce the bug?

'use strict';
const zlib = require('node:zlib');

function compress(create, writes) {
  return new Promise((resolve, reject) => {
    const stream = create();
    const chunks = [];
    stream.on('data', (chunk) => chunks.push(chunk));
    stream.on('end', () => resolve(Buffer.concat(chunks)));
    stream.on('error', reject);
    for (const w of writes) stream.write(w);
    stream.end();
  });
}

(async () => {
  const oneWrite = await compress(zlib.createZstdCompress, ['hello world']);
  const twoWrites = await compress(zlib.createZstdCompress, ['hello ', 'world']);

  console.log('one write: ', oneWrite.toString('hex'));
  console.log('two writes:', twoWrites.toString('hex'));
  console.log('extra bytes:', twoWrites.subarray(oneWrite.length).toString('hex'));

  // Same input, queued writes: gzip and brotli are unaffected.
  for (const [name, create, decompress] of [
    ['gzip', zlib.createGzip, zlib.gunzipSync],
    ['brotli', zlib.createBrotliCompress, zlib.brotliDecompressSync],
  ]) {
    const a = await compress(create, ['hello world']);
    const b = await compress(create, ['hello ', 'world']);
    console.log(name, 'lengths', a.length, b.length, decompress(b).toString());
  }
})();

How often does it reproduce? Is there a required condition?

Every time end() is called while at least one write is still queued in the compressor. In practice that is most pipelines whose source ends quickly, e.g. pipeline(tar.create(...), zlib.createZstdCompress(), fs.createWriteStream(...)): every archive we packed that way had the extra frame.

What is the expected behavior? Why is that the expected behavior?

One zstd frame, the same bytes whether the input arrived in one write or several:

one write:  28b52ffd005859000068656c6c6f20776f726c64
two writes: 28b52ffd005859000068656c6c6f20776f726c64

That's what gzip and brotli do. How the input was split into writes shouldn't change the compressed output.

What do you see instead?

one write:  28b52ffd005859000068656c6c6f20776f726c64
two writes: 28b52ffd005859000068656c6c6f20776f726c6428b52ffd2000010000
extra bytes: 28b52ffd2000010000
gzip lengths 31 31 hello world
brotli lengths 15 15 hello world

A second, empty zstd frame (28b52ffd 20 00 01 00 00: magic, single-segment descriptor with content size 0, one empty raw last block) is appended.

Additional information

I think the cause is in lib/zlib.js:

• In ZlibBase#_transform, when this.writableEnded && this.writableLength === chunk.byteLength, the last queued chunk is processed with _finishFlushFlag, which ends the frame.
• ZlibBase#_flush then calls _transform again with an empty buffer. writableEnded is still true and writableLength is 0, so that call gets the finish flag too.

For deflate and brotli a second finish on a finished stream emits nothing. For zstd, ZSTD_compressStream2(..., ZSTD_e_end) on a context whose frame has just completed starts and completes a new frame. ZstdCompressContext::DoThreadPoolWork in src/node_zlib.cc doesn't guard against that. When end() is called with nothing queued, the frame is only finished once, so the output is a single frame.

The extra frame is valid zstd, but it has real consequences:

1. The output depends on stream timing, not content. We hash the archive for deduplication, so identical inputs can hash differently.
2. In released versions (including v26.9.0), createZstdDecompress throws Unknown frame descriptor when a read chunk boundary falls inside those 9 bytes. With fs.createReadStream's default 64 KiB chunks, that's about 1 archive in 8,200, and it fails the same way every time. We hit this in production on an archive that zstd -t and zstdDecompressSync both accept. I believe #65865 fixes the decoding side on main, but the compressor still writes the extra frame.

Our workaround is to strip a trailing 28b52ffd2000010000 after compressing.

提供された JavaScript の再現から始め、次に lib/zlib.js、特に ZlibBase#_transform と _flush を読みます。src/node_zlib.cc の ZstdCompressContext::DoThreadPoolWork を調査し、キューに入れられた書き込みの経路を gzip と brotli の動作と比較します。createZstdCompress が 1 回または複数回の書き込みに対して同一のバイト列を持つ 1 つのフレームを出力し、末尾に空のフレームを出力しなければ完了です。