Hang at shutdown

Version

26.4.0

Platform

NetBSD 11.99.6/x86_64

Subsystem

No response

What steps will reproduce the bug?

When building chromium in pkgsrc, nodejs is used as a build tool. I have more than once seen a nodejs process hang during this.
Today I've accessed the hanging process and extracted a backtrace.

How often does it reproduce? Is there a required condition?

Twice in the last week (of about a chromium build a day).

What is the expected behavior? Why is that the expected behavior?

Clean shutdown.

What do you see instead?

The nodejs process does not exit. Here's the backtrace:

(gdb) thread apply all bt

Thread 3 (LWP 22392 of process 22392 "node-MainThread"):
#0  0x00007018c6e40dea in _lwp_wait () from /usr/lib/libc.so.12
#1  0x00007018c7275d88 in pthread_join (thread=0x7018c4b12000, valptr=valptr@entry=0x0) at /usr/src/lib/libpthread/pthread.c:718
#2  0x00007018c85c265e in uv_thread_join (tid=<optimized out>) at src/unix/thread.c:295
#3  0x00000000011cbd8a in node::WorkerThreadsTaskRunner::Shutdown (this=0x7018c4d168d0) at /usr/include/g++/bits/unique_ptr.h:193
#4  0x00000000011cccce in node::NodePlatform::Shutdown (this=0x7018c4cee340) at ../src/node_platform.cc:529
#5  node::NodePlatform::Shutdown (this=0x7018c4cee340) at ../src/node_platform.cc:526
#6  0x0000000001039db9 in node::V8Platform::Dispose (this=<optimized out>) at ../src/node_v8_platform-inl.h:95
#7  node::DisposePlatform () at ../src/node_v8_platform-inl.h:178
#8  node::DefaultProcessExitHandlerInternal (env=<optimized out>, exit_code=node::ExitCode::kNoFailure) at ../src/api/environment.cc:1155
#9  0x00000000010a58d2 in std::function<void(node::Environment*, node::ExitCode)>::operator() (this=0x7018c3d98bb8, __args#0=0x7018c3d98000, __args#1=node::ExitCode::kNoFailure) at /usr/include/g++/bits/std_function.h:59
1
#10 node::Environment::Exit (this=0x7018c3d98000, exit_code=node::ExitCode::kNoFailure) at ../src/env.cc:1930
#11 0x00007f7ff774fae9 in ?? ()
#12 0x00007018c3d464e9 in ?? ()
#13 0x0000000000000001 in ?? ()
#14 0x00007f7fff8bbc90 in ?? ()
#15 0x0000000000000024 in ?? ()
#16 0x00007f7fff8bbd48 in ?? ()
#17 0x00007f7ff774e0a4 in ?? ()
#18 0x00007018c4a04000 in ?? ()
#19 0x00007018c3f00011 in ?? ()
#20 0x00007018c282c181 in ?? ()
#21 0x00007018c28061a1 in ?? ()
#22 0x00007018c2838339 in ?? ()
#23 0x0000000000000000 in ?? ()


Thread 2 (LWP 6025 of process 22392 "node-V8Worker"):
#0  0x00007018c6eb5a6a in ___lwp_park60 () from /usr/lib/libc.so.12
#1  0x00007018c7273d3d in pthread_cond_timedwait (cond=0x7018c4c47770, mutex=0x7018c4c47740, abstime=0x0) at /usr/src/lib/libpthread/pthread_cond.c:178
#2  0x00000000026b24e3 in absl::synchronization_internal::PthreadWaiter::Wait (this=this@entry=0x7018c4c47740, t=..., t@entry=...) at ../deps/v8/third_party/abseil-cpp/absl/synchronization/internal/pthread_waiter.cc:117
#3  0x00000000026b22a9 in AbslInternalPerThreadSemWait (t=...) at ../deps/v8/third_party/abseil-cpp/absl/synchronization/internal/per_thread_sem.cc:93
#4  0x000000000269f7b3 in absl::CondVar::WaitCommon (this=this@entry=0x7018c4ca84d0, mutex=mutex@entry=0x7018c4ca84c8, t=..., t@entry=...) at ../deps/v8/third_party/abseil-cpp/absl/synchronization/mutex.cc:2641
#5  0x00000000026d4c89 in absl::CondVar::Wait (this=0x7018c4ca84d0, mu=0x7018c4ca84c8) at ../deps/v8/third_party/abseil-cpp/absl/synchronization/mutex.h:989
#6  0x0000000001a9f776 in operator() (__closure=<synthetic pointer>) at ../deps/v8/src/heap/collection-barrier.cc:166
#7  v8::internal::LocalHeap::ParkAndExecuteCallback<v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda()> > (this=0x7018c2dcf688, callback=...) at
 ../deps/v8/src/heap/local-heap-inl.h:71
#8  operator() (__closure=<optimized out>) at ../deps/v8/src/heap/local-heap-inl.h:88
#9  heap::base::Stack::SetMarkerForBackgroundThreadAndCallbackImpl<v8::internal::LocalHeap::ExecuteWhileParked<v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda()> >(v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda()>)::<lambda()> >(heap::base::Stack *, void *, const void *) (stack=0x7018c4ca4760, argument=<optimized out>, stack_end=<optimized out>) at ../deps/v8/src/heap/base/stack.h:208
#10 0x00000000013d0f59 in PushAllRegistersAndIterateStack ()
#11 0x0000000001aa0085 in heap::base::Stack::SetMarkerForBackgroundThreadAndCallback<v8::internal::LocalHeap::ExecuteWhileParked<v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda()> >(v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda()>)::<lambda()> > (this=0x7018c4ca4760, thread=<optimized out>, callback=...) at ../deps/v8/src/heap/base/stack.h:90
#12 v8::internal::LocalHeap::ExecuteWithStackMarker<v8::internal::LocalHeap::ExecuteWhileParked<v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda
()> >(v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda()>)::<lambda()> > (this=0x7018c2dcf688, callback=...) at ../deps/v8/src/heap/local-heap-i
nl.h:80
#13 v8::internal::LocalHeap::ExecuteWhileParked<v8::internal::CollectionBarrier::AwaitCollectionBackground(v8::internal::LocalHeap*, v8::internal::RequestedGCKind)::<lambda()> > (this=0x7018c2dcf688, callback=...) at ../deps/v8/src/heap/local-heap-inl.h:87
#14 v8::internal::CollectionBarrier::AwaitCollectionBackground (this=0x7018c4ca84c0, local_heap=0x7018c2dcf688, kind=<optimized out>) at ../deps/v8/src/heap/collection-barrier.cc:158
#15 0x0000000001b24475 in v8::internal::HeapAllocator::CollectGarbageAndRetryAllocation (this=this@entry=0x7018c2dcf730, allocate=..., allocation=allocation@entry=v8::internal::AllocationType::kCode) at ../deps/v8/src/heap/heap-allocator.cc:543
#16 0x0000000001b24664 in v8::internal::HeapAllocator::RetryCustomAllocateLight (this=0x7018c2dcf730, allocate=..., allocation=v8::internal::AllocationType::kCode) at ../deps/v8/src/heap/heap-allocator.cc:518
#17 0x0000000001b24721 in v8::internal::HeapAllocator::AllocateRawSlowPath (this=this@entry=0x7018c2dcf730, retry_mode=retry_mode@entry=v8::internal::HeapAllocator::kLightRetry, size=size@entry=1664, allocation=allocation@entry=v8::internal::AllocationType::kCode, origin=origin@entry=v8::internal::AllocationOrigin::kRuntime, alignment=alignment@entry=v8::internal::kTaggedAligned, hint=hint@entry=...) at ../deps/v8/src/heap/heap-allocato
r.cc:215
#18 0x0000000001afa11a in v8::internal::HeapAllocator::AllocateRawWith<(v8::internal::HeapAllocator::AllocationRetryMode)0> (this=0x7018c2dcf730, size=1664, allocation=v8::internal::AllocationType::kCode, origin=v8::internal::AllocationOrigin::kRuntime, alignment=v8::internal::kTaggedAligned, hint=...) at ../deps/v8/src/heap/allocation-result.h:41
#19 v8::internal::LocalHeap::AllocateRawWith<(v8::internal::HeapAllocator::AllocationRetryMode)0> (this=0x7018c2dcf688, object_size=1664, type=v8::internal::AllocationType::kCode, origin=v8::internal::AllocationOrigin::k
Runtime, alignment=v8::internal::kTaggedAligned, hint=...) at ../deps/v8/src/heap/local-heap-inl.h:44
#20 v8::internal::Factory::CodeBuilder::AllocateUninitializedInstructionStream (this=this@entry=0x7018c2dcf280, retry_allocation_or_fail=retry_allocation_or_fail@entry=false) at ../deps/v8/src/heap/factory.cc:285
#21 0x0000000001b12c6a in v8::internal::Factory::CodeBuilder::BuildInternal (this=this@entry=0x7018c2dcf280, retry_allocation_or_fail=retry_allocation_or_fail@entry=false) at ../deps/v8/src/heap/factory.cc:150
#22 0x0000000001b1348a in v8::internal::Factory::CodeBuilder::TryBuild (this=this@entry=0x7018c2dcf280) at ../deps/v8/src/heap/factory.cc:295
#23 0x00000000020f58bd in v8::internal::baseline::BaselineCompiler::Build (this=this@entry=0x7018c2dcf3c0) at ../deps/v8/src/baseline/baseline-compiler.cc:359
#24 0x00000000020f25af in v8::internal::baseline::BaselineCompilerTask::Compile (this=0x7018a798a800, local_isolate=0x7018c2dcf680) at ../deps/v8/src/execution/local-isolate.h:61
#25 v8::internal::baseline::BaselineBatchCompilerJob::Compile (this=0x7018c3f53ce0, local_isolate=0x7018c2dcf680) at ../deps/v8/src/baseline/baseline-batch-compiler.cc:129
#26 v8::internal::baseline::ConcurrentBaselineCompiler::JobDispatcher::Run (this=<optimized out>, delegate=<optimized out>) at ../deps/v8/src/baseline/baseline-batch-compiler.cc:171
#27 0x00000000025f1f71 in v8::platform::DefaultJobWorker::Run (this=0x7018c3f53180) at ../deps/v8/src/libplatform/default-job.h:147
#28 0x00000000011cc407 in node::(anonymous namespace)::PlatformWorkerThread (data=0x7018c4cd7820) at ../src/node_platform.cc:80
#29 0x00007018c7275383 in pthread__create_tramp (cookie=0x7018c4b12000) at /usr/src/lib/libpthread/pthread.c:601
#30 0x00007018c6e96de0 in ?? () from /usr/lib/libc.so.12
#31 0x0000000000000000 in ?? ()

Thread 1 (LWP 10116 of process 22392 "SignalInspector"):
#0  0x00007018c6e40f6a in _ksem_wait () from /usr/lib/libc.so.12
#1  0x00007018c85c2cc4 in uv__sem_wait (sem=0x491bf80 <node::inspector::(anonymous namespace)::start_io_thread_semaphore>) at src/unix/thread.c:660
#2  uv_sem_wait (sem=0x491bf80 <node::inspector::(anonymous namespace)::start_io_thread_semaphore>) at src/unix/thread.c:716
#3  0x00000000012f9dff in node::inspector::(anonymous namespace)::StartIoThreadMain (unused=<optimized out>) at ../src/inspector_agent.cc:94
#4  0x00007018c7275383 in pthread__create_tramp (cookie=0x7018c4b15800) at /usr/src/lib/libpthread/pthread.c:601
#5  0x00007018c6e96de0 in ?? () from /usr/lib/libc.so.12
#6  0x0000000000000000 in ?? ()

It looks like one thread is calling pthread_join, while the thread it wants to join is hanging in an unrestricted pthread_cond_timedwait (abstime = 0x0).

Probably the shutdown needs to first wake up the waiting thread.

Additional information

No response

Beginne mit src/node_platform.cc, insbesondere mit WorkerThreadsTaskRunner::Shutdown und NodePlatform::Shutdown, und untersuche anschließend den V8 worker stack sowie inspector_agent.cc, die im Backtrace angezeigt werden. Reproduziere den Shutdown-Hänger beim Bauen von Chromium auf NetBSD und verfolge den Aufweckpfad des wartenden Threads; abgeschlossen ist die Arbeit, wenn der Prozess sauber beendet wird, ohne in pthread_join hängen zu bleiben.