nodejs / nodejs/node

zlib: one-shot gzip()/deflate() results accumulate in arrayBuffers until OOM — GC never prompted (regression in v24.15.0)

Aperta
#65,600 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Lingua principale
JavaScript
Stelle
122k
Fork
37.3k
Merge medio
4g 2h
PR unite (30g)
283

Descrizione

Version

v24.15.0 through v24.20.0 (first bad release: v24.15.0; still present on v24.19.0 and v24.20.0, the latest 24.x at time of filing). Confirmed clean: v22.12.0, v22.22.0, v24.0.0, v24.13.0, v24.14.0.

Platform
Darwin 25.5.0 arm64

Also reproduced in production on Linux x64 (containerized, cgroup v2) — that is where we hit OOMKills.

Subsystem

zlib

What steps will reproduce the bug?

Call the one-shot convenience API (zlib.gzip()zlib.deflate() behaves identically) in a loop and watch process.memoryUsage().arrayBuffers. No dependencies — save as repro.js:

"use strict";
// One-shot zlib.gzip() in a loop; watch process.memoryUsage().arrayBuffers.
// Run:            node repro.js
// With forced GC: FORCE_GC=1 node --expose-gc repro.js
const zlib = require("node:zlib");

const N = 2000;
// ~120 KB compressible payload (repeating structured JSON)
const chunk = Buffer.from(
  JSON.stringify({ resourceSpans: [{ scopeSpans: [{ spans: [{ name: "POST /x", attrs: { a: 1 } }] }] }] }),
);
const payload = Buffer.concat(Array(Math.ceil((120 * 1024) / chunk.length)).fill(chunk));

const mb = (x) => (x / 1048576).toFixed(1);
function sample(label) {
  if (process.env.FORCE_GC === "1" && global.gc) global.gc();
  const m = process.memoryUsage();
  console.log(
    `${label}\theapUsed=${mb(m.heapUsed)}\texternal=${mb(m.external)}\tarrayBuffers=${mb(m.arrayBuffers)}`,
  );
}

console.log(`node=${process.version} force_gc=${process.env.FORCE_GC === "1"}`);
sample("start");
let i = 0;
(function loop() {
  if (i >= N) return setTimeout(() => sample("end"), 200);
  i++;
  if (i % 500 === 0) sample(`i=${i}`);
  zlib.gzip(payload, (err) => {
    if (err) throw err;
    loop();
  });
})();
How often does it reproduce? Is there a required condition?

Every run, immediately. Two conditions:

  1. The one-shot API (zlib.gzip, zlib.deflate, …). The stream classes are NOT affected — a loop creating a createGunzip() stream per message stays flat, presumably because Close() frees native state deterministically.
  2. A JS heap comfortable enough that V8 does not run major GCs on its own. That is exactly the production steady state of a typical server: our pods sat at ~150 MiB heapUsed against a 432 MiB old-space limit, so nothing ever collected the dead results.
What is the expected behavior? Why is that the expected behavior?

The memory retained by completed one-shot calls should count as GC pressure, so V8 collects the dead result buffers before they accumulate meaningfully. That is what every release up to and including v24.14.0 does — the same loop is flat (sawtooths back down without any forced GC):

=== v24.14.0 ===
node=v24.14.0 force_gc=false
start	heapUsed=3.9	external=1.5	arrayBuffers=0.1
i=500	heapUsed=4.2	external=1.7	arrayBuffers=0.4
i=1000	heapUsed=4.4	external=2.1	arrayBuffers=0.7
i=1500	heapUsed=4.5	external=2.2	arrayBuffers=0.8
i=2000	heapUsed=4.2	external=1.7	arrayBuffers=0.3
end	heapUsed=4.3	external=1.7	arrayBuffers=0.3
What do you see instead?

From v24.15.0 on, arrayBuffers grows linearly with the number of calls and never comes back down:

=== v24.20.0 ===
node=v24.20.0 force_gc=false
start	heapUsed=4.2	external=1.9	arrayBuffers=0.3
i=500	heapUsed=5.2	external=12.2	arrayBuffers=10.6
i=1000	heapUsed=6.9	external=22.6	arrayBuffers=21.0
i=1500	heapUsed=7.2	external=33.1	arrayBuffers=31.4
i=2000	heapUsed=8.5	external=43.5	arrayBuffers=41.9
end	heapUsed=8.5	external=43.5	arrayBuffers=41.9

End-state arrayBuffers across versions (same script, 2000 iterations):

version end arrayBuffers
v22.12.0 2.7 MiB ✅
v24.14.0 0.3 MiB ✅
v24.15.0 31.5 MiB ❌
v24.19.0 41.9 MiB ❌
v24.20.0 41.9 MiB ❌

The buffers are not unreclaimable: with FORCE_GC=1 node --expose-gc repro.js every affected version ends at ~0.3 MiB. The memory is ordinary garbage that nothing ever prompts V8 to collect — which is why the growth is unbounded in a long-running server whose heap is otherwise comfortable.

Growth granularity is the compressed output rounded up to the 16 KB default chunkSize: this compressible 120 KB payload leaks ~16 KB/call; a 16 KB random (incompressible) payload leaks ~31 KB/call; 64 KB random leaks ~94 KB/call.

Additional information

Release-level bisect points at v24.15.0, and the only zlib change in that release is #61717 ("src: refactor compression allocation tracking, enable for zstd"), i.e. commits bef661f182, 3c8f700fd7, 94dbb36d4d, e8079a8297. The behavior is consistent with the one-shot path's retained allocations no longer being reported to V8's external-memory GC-pressure accounting after that refactor.

Real-world impact: @grpc/grpc-js runs zlib.gzip(message, cb) once per outgoing message when compression is enabled (compression-filter.js), so any service exporting OpenTelemetry traces over OTLP/gRPC with CompressionAlgorithm.GZIP leaks per export. After upgrading a production service from Node 22.12 to 24.19 with no dependency changes, its pods accumulated arrayBuffers at ~3 MiB/min and were OOMKilled roughly every 90 minutes; heapUsed stayed flat the whole time. Disabling exporter compression fully mitigates it.

Searched for existing reports before filing: the closest hits are the Blob.stream()/CompressionStream RSS leaks (#64105, #63574, #63708), but those leak RSS rather than arrayBuffers, begin at v24.16 or v26, and are not reclaimed the same way — this one starts exactly at v24.15.0, is fully reclaimed by forced GC, and needs only plain zlib.gzip().

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia eseguendo repro.js con le versioni di Node.js interessate e quelle corrette, quindi esamina il percorso zlib one-shot e i quattro commit di #61717 elencati nella issue. Traccia il modo in cui le allocazioni completate vengono segnalate a V8 e verifica che il fix ripristini la pressione del GC senza influire sulle classi stream; la riproduzione non dovrebbe più accumulare arrayBuffers senza GC forzato.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
javascript, node.js
Ambito
backend, performance
Tipo di issue
Bug
Difficoltà
4/5
Tempo stimato
3-5 giorni
Stato di attività
Attiva
Chiarezza
Abbastanza chiara
Idoneità per principianti
58/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.