Process list incomplete after reboot

What's going wrong?

I have several Node.js processes managed by pm2, and I've installed the default systemd startup script. When I reboot the machine, pm2 starts up with an incomplete list of processes.

Related to https://github.com/Unitech/pm2/issues/3865.

How could we reproduce this issue?

I managed to narrow down the issue to the following factors:

1. I have one process from the many that takes a long time to shut down and exit (let's say 100 seconds)
2. During the shutdown the systemd service will time out after 90 seconds by default (TimeoutStopUSec=1min 30s)
3. This will cause pm2 to dump the process list in an incomplete state (The following line appears in the logs after 90 seconds: pm2 has been killed by signal, dumping process list before exit...)

I believe #3865 was supposed to solve this issue, but apparently it didn't cover all the cases. For my use-case it'd be nice to have the option to never dump the process list during a shutdown.

Supporting information

I tried with the latest 5.4.2 version of pm2 on Ubuntu 24.04 LTS with Node.js 20.16.0.

--- PM2 report ----------------------------------------------------------------
Date                 : Mon Aug 12 2024 11:40:26 GMT+0200 (Central European Summer Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 5.4.2
node version         : 20.16.0
node path            : not found
argv                 : /home/gera/.nvm/versions/node/v20.16.0/bin/node,/home/gera/.nvm/versions/node/v20.16.0/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : gera
uid                  : 501
gid                  : 501
uptime               : 22min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 5.4.2
node version         : 20.16.0
node path            : /home/gera/.nvm/versions/node/v20.16.0/bin/pm2
argv                 : /home/gera/.nvm/versions/node/v20.16.0/bin/node,/home/gera/.nvm/versions/node/v20.16.0/bin/pm2,report
argv0                : node
user                 : gera
uid                  : 501
gid                  : 501
===============================================================================
--- System info --------------------------------------------
arch                 : arm64
platform             : linux
type                 : Linux
cpus                 : unknown
cpus nb              : 10
freemem              : 15780114432
totalmem             : 16742363136
home                 : /home/gera
===============================================================================
--- PM2 list -----------------------------------------------
┌────┬───────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name      │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├────┼───────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0  │ server    │ default     │ 1.0.0   │ fork    │ 1027     │ 19m    │ 1    │ online    │ 0.8%     │ 49.3mb   │ gera     │ disabled │
└────┴───────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/home/gera/.pm2/pm2.log last 20 lines:
PM2        | 2024-08-12T11:21:17: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:17: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:17: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:17: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:18: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:18: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:18: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:18: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:18: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:18: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:18: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:18: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:18: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:18: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:19: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:19: PM2 log: pid=995 msg=failed to kill - retrying in 100ms
PM2        | 2024-08-12T11:21:19: PM2 log: App [server:0] exited with code [0] via signal [SIGINT]
PM2        | 2024-08-12T11:21:19: PM2 log: pid=995 msg=process killed
PM2        | 2024-08-12T11:21:19: PM2 log: App [server:0] starting in -fork mode-
PM2        | 2024-08-12T11:21:19: PM2 log: App [server:0] online

Start with the default systemd startup script and the shutdown path that emits “pm2 has been killed by signal, dumping process list before exit...”. Reproduce the 90-second timeout with a process that takes about 100 seconds to exit, then verify that shutdown no longer overwrites the complete process list when the requested behavior is configured.