Resurrect has app in stopped state

What's going wrong?

After a reboot (perhaps not a clean reboot), the process list is resurrected properly, though all apps remain in stopped state. pm2 save was definitely not called in between and was saved in a running state. Restart counts are all at zero with no error indicated or logged on startup.

How could we reproduce this issue?

I haven't been able to reproduce this consistently and it happens seldomly on units remotely, so it is difficult to monitor.

Supporting information

Happens on latest version as welll

--- PM2 report ----------------------------------------------------------------
Date                 : Tue Nov 10 2020 07:12:12 GMT+0000 (Coordinated Universal Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 4.5.0
node version         : 13.7.0
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : jericho
uid                  : 1000
gid                  : 1000
uptime               : 13min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 4.5.0
node version         : 13.7.0
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : jericho
uid                  : 1000
gid                  : 1000
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Atom(TM) x5-Z8350  CPU @ 1.44GHz
cpus nb              : 4
freemem              : 185622528
totalmem             : 3517788160
home                 : /home/jericho
===============================================================================
--- PM2 list -----------------------------------------------
┌─────┬──────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id  │ name             │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├─────┼──────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 1   │ secuvue-agent    │ default     │ N/A     │ fork    │ 7861     │ 12m    │ 0    │ online    │ 101.2%   │ 644.5mb  │ jericho  │ disabled │
└─────┴──────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘
Module
┌────┬───────────────────────────────────────┬────────────────────┬───────┬──────────┬──────┬──────────┬──────────┬──────────┐
│ id │ module                                │ version            │ pid   │ status   │ ↺    │ cpu      │ mem      │ user     │
├────┼───────────────────────────────────────┼────────────────────┼───────┼──────────┼──────┼──────────┼──────────┼──────────┤
│ 0  │ pm2-logrotate                         │ 2.6.0              │ 7666  │ online   │ 0    │ 0.7%     │ 44.7mb   │ jericho  │
└────┴───────────────────────────────────────┴────────────────────┴───────┴──────────┴──────┴──────────┴──────────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/home/jericho/.pm2/pm2.log last 20 lines:
PM2        | 2020-11-10T06:59:13: PM2 log: ===============================================================================
PM2        | 2020-11-10T06:59:13: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
PM2        | 2020-11-10T06:59:13: PM2 log: Time                 : Tue Nov 10 2020 06:59:13 GMT+0000 (Coordinated Universal Time)
PM2        | 2020-11-10T06:59:13: PM2 log: PM2 version          : 4.5.0
PM2        | 2020-11-10T06:59:13: PM2 log: Node.js version      : 13.7.0
PM2        | 2020-11-10T06:59:13: PM2 log: Current arch         : x64
PM2        | 2020-11-10T06:59:13: PM2 log: PM2 home             : /home/jericho/.pm2
PM2        | 2020-11-10T06:59:13: PM2 log: PM2 PID file         : /home/jericho/.pm2/pm2.pid
PM2        | 2020-11-10T06:59:13: PM2 log: RPC socket file      : /home/jericho/.pm2/rpc.sock
PM2        | 2020-11-10T06:59:13: PM2 log: BUS socket file      : /home/jericho/.pm2/pub.sock
PM2        | 2020-11-10T06:59:13: PM2 log: Application log path : /home/jericho/.pm2/logs
PM2        | 2020-11-10T06:59:13: PM2 log: Worker Interval      : 30000
PM2        | 2020-11-10T06:59:13: PM2 log: Process dump file    : /home/jericho/.pm2/dump.pm2
PM2        | 2020-11-10T06:59:13: PM2 log: Concurrent actions   : 2
PM2        | 2020-11-10T06:59:13: PM2 log: SIGTERM timeout      : 1600
PM2        | 2020-11-10T06:59:13: PM2 log: ===============================================================================
PM2        | 2020-11-10T06:59:13: PM2 log: App [pm2-logrotate:0] starting in -fork mode-
PM2        | 2020-11-10T06:59:13: PM2 log: App [pm2-logrotate:0] online
PM2        | 2020-11-10T06:59:22: PM2 log: App [secuvue-agent:1] starting in -fork mode-
PM2        | 2020-11-10T06:59:22: PM2 log: App [secuvue-agent:1] online

Note this report is after I manually called pm2 restart all. As a precaution I did a pm2 unstartup followed by pm2 startup as well.

Start with the PM2 startup and resurrect flow, including the saved process state in ~/.pm2/dump.pm2, and compare it with the daemon logs shown in the report. Investigate what happens during reboot when apps saved as running are restored, ideally using the reported commands and environment. Done means saved apps reliably return online without requiring pm2 restart all.