Pm2 Daemon crashes and doesn't recover

Start with the reported PM2 5.2.0 environment: Node.js 16.18.1 on RHEL 9.2, and the daemon log showing a JavaScript heap out-of-memory crash. Inspect the daemon startup and recovery paths, then reproduce the failure if possible. Done means the daemon no longer remains unavailable after this crash, with a regression check for the reported behavior.

What's going wrong?

PM2 Daemon is crashing and is not recovering

Supporting information

:/.pm2=>node -v
v16.18.1
:/.pm2=>cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="9.2 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.2 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"

<--- Last few GCs --->

[148024:0x55f9bfd704d0] 279067495 ms: Mark-sweep 4063.2 (4142.9) -> 4061.1 (4142.9) MB, 3675.7 / 0.0 ms  (average mu = 0.140, current mu = 0.004) allocation failure scavenge might not succee
d
[148024:0x55f9bfd704d0] 279071596 ms: Mark-sweep 4063.1 (4142.9) -> 4061.1 (4143.1) MB, 4079.9 / 0.0 ms  (average mu = 0.075, current mu = 0.005) allocation failure scavenge might not succee
d


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x7fdbaef0a6cc node::Abort() [/lib64/libnode.so.93]
 2: 0x7fdbaec55132 node::OOMErrorHandler(char const*, bool) [/lib64/libnode.so.93]
 3: 0x7fdbaf51c6a4 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/lib64/libnode.so.93]
 4: 0x7fdbaf51c962 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/lib64/libnode.so.93]
 5: 0x7fdbaf6b71b9  [/lib64/libnode.so.93]
 6: 0x7fdbaf6b729d  [/lib64/libnode.so.93]
 7: 0x7fdbaf6cfa3e v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/lib64/libnode.so.93]
 8: 0x7fdbaf6d0411 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/lib64/libnode.so.93]
 9: 0x7fdbaf6d32fa v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/lib64/libnode.so.93]
10: 0x7fdbaf6d337b v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/lib64/libnode.so.93]
11: 0x7fdbaf699891 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/lib64/libnode.so.93]
12: 0x7fdbaf9c16bd v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/lib64/libnode.so.93]
13: 0x7fdbaf177b39  [/lib64/libnode.so.93]
2024-11-14T10:07:19: PM2 log: ===============================================================================
2024-11-14T10:07:19: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2024-11-14T10:07:19: PM2 log: Time                 : Thu Nov 14 2024 10:07:19 GMT+0530 (India Standard Time)
2024-11-14T10:07:19: PM2 log: PM2 version          : 5.2.0
2024-11-14T10:07:19: PM2 log: Node.js version      : 16.18.1
2024-11-14T10:07:19: PM2 log: Current arch         : x64
2024-11-14T10:07:19: PM2 log: PM2 home             : /home/k2/.pm2
2024-11-14T10:07:19: PM2 log: PM2 PID file         : /home/k2/.pm2/pm2.pid
2024-11-14T10:07:19: PM2 log: RPC socket file      : /home/k2/.pm2/rpc.sock
2024-11-14T10:07:19: PM2 log: BUS socket file      : /home/k2/.pm2/pub.sock
2024-11-14T10:07:19: PM2 log: Application log path : /home/k2/.pm2/logs
2024-11-14T10:07:19: PM2 log: Worker Interval      : 30000
2024-11-14T10:07:19: PM2 log: Process dump file    : /home/k2/.pm2/dump.pm2
2024-11-14T10:07:19: PM2 log: Concurrent actions   : 2