Cluster Mode - All Clusters restarting on error

What's going wrong?

I am running a NodeJS application in Cluster mode: pm2 start server.js -i max

Assume I have 2 clusters (0 and 1)... Our application is a simple NodeJS backend app running Express. However, there is the occasion where there is an unhandled error that causes the app to go down (thus PM2 needs to restart). I am noticing that whenever PM2 restarts on cluster mode, however, it brings ALL clusters down temporarily to restart them ALL...but I simply need the worker thread to restart, not all of the threads that did not error.

Our app needs a few seconds to "reboot" before it can accept connections (needs to connect to DB). Therefore, we don't want the restarting cluster to be 'online' until about 30 seconds after the restart. We tried using --listen-timeout 30000 but the issue still appears to be that all clusters are restarting.

How could we reproduce this issue?

• Start a Express (node.js) app using pm2 cluster mode (pm2 start server.js -i max)
• Trigger a error that crashes the instance
• Run pm2 status and you will notice that all instances in the cluster are restarted (their restart value increases by 1)

Supporting information

• PM2 version 5.3.0 (We were originally on PM2 5.2.0 but the issue was persistent in both versions)
• Running on Node 18.7.0

$ pm2 report
--- PM2 report ----------------------------------------------------------------
Date                 : Wed May 10 2023 13:45:59 GMT+0000 (Coordinated Universal Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 5.3.0
node version         : 18.7.0
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : nodejs
uid                  : 997
gid                  : 997
uptime               : 2534min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 5.3.0
node version         : 18.7.0
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : nodejs
uid                  : 997
gid                  : 997
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : DO-Premium-Intel
cpus nb              : 2
freemem              : 1164423168
totalmem             : 2079416320
home                 : /home/nodejs
===============================================================================
--- PM2 list -----------------------------------------------
┌────┬────────────────────┬──────────┬──────┬───────────┬──────────┬──────────┐
│ id │ name               │ mode     │ ↺    │ status    │ cpu      │ memory   │
├────┼────────────────────┼──────────┼──────┼───────────┼──────────┼──────────┤
│ 0  │ server             │ cluster  │ 6    │ online    │ 0.2%     │ 216.5mb  │
│ 1  │ server             │ cluster  │ 6    │ online    │ 0.2%     │ 220.0mb  │
└────┴────────────────────┴──────────┴──────┴───────────┴──────────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/home/nodejs/.pm2/pm2.log last 20 lines:
PM2        | 2023-05-09T17:44:10: PM2 log: App [server:_old_0] exited with code [0] via signal [SIGINT]
PM2        | 2023-05-09T17:44:10: PM2 log: pid=1342160 msg=process killed
PM2        | 2023-05-09T17:44:10: PM2 log: App [server:1] starting in -cluster mode-
PM2        | 2023-05-09T17:44:10: PM2 log: App [server:1] online
PM2        | 2023-05-09T17:44:13: PM2 log: Stopping app:server id:_old_1
PM2        | 2023-05-09T17:44:13: PM2 log: App name:server id:_old_1 disconnected
PM2        | 2023-05-09T17:44:13: PM2 log: App [server:_old_1] exited with code [0] via signal [SIGINT]
PM2        | 2023-05-09T17:44:13: PM2 log: pid=1342172 msg=process killed
PM2        | 2023-05-09T18:30:22: PM2 log: App [server:0] starting in -cluster mode-
PM2        | 2023-05-09T18:30:22: PM2 log: App [server:0] online
PM2        | 2023-05-09T18:30:25: PM2 log: Stopping app:server id:_old_0
PM2        | 2023-05-09T18:30:25: PM2 log: App name:server id:_old_0 disconnected
PM2        | 2023-05-09T18:30:25: PM2 log: App [server:_old_0] exited with code [0] via signal [SIGINT]
PM2        | 2023-05-09T18:30:25: PM2 log: pid=1350348 msg=process killed
PM2        | 2023-05-09T18:30:25: PM2 log: App [server:1] starting in -cluster mode-
PM2        | 2023-05-09T18:30:25: PM2 log: App [server:1] online
PM2        | 2023-05-09T18:30:28: PM2 log: Stopping app:server id:_old_1
PM2        | 2023-05-09T18:30:28: PM2 log: App name:server id:_old_1 disconnected
PM2        | 2023-05-09T18:30:28: PM2 log: App [server:_old_1] exited with code [0] via signal [SIGINT]
PM2        | 2023-05-09T18:30:28: PM2 log: pid=1350360 msg=process killed

Reproduce with pm2 start server.js -i max, trigger an error, and inspect pm2 status and pm2 report. Start by tracing PM2's cluster-mode restart handling; done means the failed worker restarts without incrementing the restart counts or temporarily restarting unaffected workers, while the reported listen-timeout behavior remains testable.