nodejs / nodejs/node

v22.x: main thread self-deadlocks when creating a CpuProfiler under GC pressure — backport request for V8 a6eaf75741 (RecursiveMutex for Logger, crbug 41497149)

Aperta
#65,140 0 commenti 0 reazioni 0 assegnatari Vedi su GitHub

Nessuno ha ancora preso questa issue.

Lingua principale
JavaScript
Stelle
122k
Fork
37.3k
Merge medio
4g 2h
PR unite (30g)
283

Descrizione

v22.x backport request: V8 a6eaf75741 ("[logging] Use RecursiveMutex for Logger") — main-thread self-deadlock when a CpuProfiler is created

Version

v22.20.0 (still present in v22.23.1 — see below)

Platform
Linux 7.0.0-22-generic #22-Ubuntu SMP PREEMPT_DYNAMIC x86_64 (Docker node:22.20-bookworm-slim)
Subsystem

deps/v8

What steps will reproduce the bug?

Create a v8::CpuProfiler (e.g. any native addon or embedder calling v8::CpuProfiler::New, or in-process profiling agents — we hit it through @sentry/profiling-node, whose N-API Init creates a profiler at require() time) in a process that already has a significant amount of compiled script code, at a moment when the heap is close to the incremental-marking trigger.

CpuProfiler::NewProfilingScopeV8FileLogger::LogCompiledFunctions() dispatches CodeCreateEvents through the Logger dispatcher, which holds a non-recursive base::Mutex while calling listeners (deps/v8/src/logging/code-events.h). ProfilerListener::CodeCreateEvent initializes script line-ends, which allocates; if that allocation crosses the incremental-marking limit, V8 starts a major GC cycle on the same thread: Heap::StartIncrementalMarkingMarkCompactCollector::StartCompactionIsolate::AllowsCodeCompaction()logger()->allows_code_compaction() → attempts to re-acquire the same mutex → the main thread deadlocks against itself, forever.

This is Chromium bug 41497149, fixed upstream in May 2024 by V8 commit a6eaf75741 ("[logging] Use RecursiveMutex for Logger", https://chromium-review.googlesource.com/c/v8/v8/+/5572952) — the commit message describes exactly this scenario. The fix landed in V8 ≥ 12.7, so Node 23/24 have it; Node 22's V8 12.4 does not, and the file is unchanged in v22.23.1 (deps/v8/src/logging/code-events.h still uses plain base::MutexGuard). Node 20/21 are equally affected but EOL.

The patch is minimal (one file, MutexRecursiveMutex in the Logger dispatcher) and V8 12.4 already ships base::RecursiveMutex, so it should float trivially onto deps/v8 in v22.x.

How often does it reproduce? Is there a required condition?

Non-deterministic by nature: the incremental-marking start has to land inside the LogCompiledFunctions walk. In production (NestJS monolith on a 2-vCPU host, addon loaded early in a large module graph, container cold-boot contention) we measured roughly 1 process start in 3-4 freezing forever. Calm restarts of the same build essentially never hit it, which makes it very painful to diagnose in the field.

What is the expected behavior?

Creating a CPU profiler never deadlocks the process.

What do you see instead?

The process freezes before binding any port, with zero output and ~0 CPU, and never crashes (so process managers keep it "online"). gdb backtrace of the frozen main thread (all other threads idle — same-thread self-deadlock):

#1  pthread_mutex_lock
#2  v8::internal::Isolate::AllowsCodeCompaction()
#3  v8::internal::MarkCompactCollector::StartCompaction(...)
#4  v8::internal::IncrementalMarking::StartMarkingMajor()
#6  v8::internal::Heap::StartIncrementalMarking(...)
#7  v8::internal::PagedSpaceAllocatorPolicy::EnsureAllocation(...)
#8  v8::internal::MainAllocator::AllocateRawSlow(...)
#11 v8::internal::FactoryBase<v8::internal::Factory>::NewFixedArray(...)
#12 v8::internal::String::CalculateLineEnds(...)
#13 v8::internal::Script::InitLineEndsInternal(...)
#14 v8::internal::Script::GetPositionInfo(...)
#16 v8::internal::SourcePosition::InliningStack(...)
#17 v8::internal::ProfilerListener::CodeCreateEvent(...)
#18 v8::internal::ExistingCodeLogger::LogExistingFunction(...)
#19 v8::internal::ExistingCodeLogger::LogCompiledFunctions(...)
#20 v8::internal::ProfilingScope::ProfilingScope(...)
#21 v8::internal::CpuProfiler::EnableLogging()
#22 v8::internal::CpuProfiler::CpuProfiler(...)
#23 v8::CpuProfiler::New(...)
#24 Init(napi_env__*, napi_value__*)  (native addon, at require() time)
#25 napi_module_register_by_symbol(...)
#28 node::binding::DLOpen(...)

strace shows the main thread parked in FUTEX_WAIT_PRIVATE on a contended mutex; every other thread is in its normal idle wait.

Additional information

Workarounds we validated/identified while root-causing this:

  • don't create profilers at startup (we made the Sentry profiling integration opt-in — also reported to Sentry: https://github.com/getsentry/sentry-javascript/issues/23178);
  • --no-compact-code-space sidesteps the re-entry (AllowsCodeCompaction short-circuits before touching the Logger mutex), at the cost of disabling code-space compaction;
  • upgrading to Node ≥ 23 removes the bug.

Node 22 is in maintenance LTS until 2027-04 and this is a silent, unrecoverable hang, which is why a backport seems worth it despite maintenance mode.

Guida per i contributori

Apri la guida per i contributori

Come iniziare

  1. Leggi tutta la issue e poi la guida ai contributi del progetto.
  2. Commenta sulla issue per dire che te ne occupi tu — evita che due persone facciano lo stesso lavoro.
  3. Fai un fork del repository e lavora su un branch.
  4. Apri una pull request che faccia riferimento al numero della issue.

Direzione di ricerca

Inizia da deps/v8/src/logging/code-events.h e confronta il dispatcher di Logger con il commit di V8 a6eaf75741. Conferma che V8 12.4 fornisca base::RecursiveMutex, quindi controlla il percorso di creazione di CpuProfiler sotto pressione di GC. Il lavoro è completato quando il backport v22.x impedisce il self-deadlock del thread principale durante la creazione di un profiler.

Scritto dal modello di indicizzazione a partire dal testo della issue.

Valutazione

Stack tecnologico
cpp, node.js
Ambito
backend
Tipo di issue
Bug
Difficoltà
3/5
Tempo stimato
1-2 giorni
Stato di attività
Tranquilla
Chiarezza
Specificata chiaramente
Idoneità per principianti
72/100

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.