Unitech / Unitech/pm2

All applications disappeared. `pm2 list` displays empty list

Open
#4,798 10 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
43.3k
Forks
2.7k
No merged PRs in 30d

Description

What's going wrong?

I used pm2 for months, and yesterday all applications disappeared. It happened suddenly - I didn't reboot my computer, didn't create any new applications - just reviewed logs. At some point, when I ran pm2 list it displayed empty list. I waited for some time (I remember early I had similar behavior right after the computer reboot and had to wait some time) - no, after few hours it's still empty.

How could we reproduce this issue?

Not sure.

Supporting information

OS: macOS Mojave v 10.14.6 (18G5033)
MacBook Pro (15-inch, 2017)
Processor 3.1 GHz Intel Core i7
Memory 16 GB 2133 MHz LPDDR3
pm2 -v: 3.2.9

$ pm2 report

--- PM2 report ----------------------------------------------------------------
Date                 : Mon Aug 03 2020 11:12:40 GMT+0300 (Eastern European Summer Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 3.2.9
node version         : 10.18.0
node path            : /usr/local/bin/pm2
argv                 : /usr/local/bin/node,/usr/local/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : max
uid                  : 501
gid                  : 20
uptime               : 869min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 3.2.9
node version         : 10.18.0
node path            : /usr/local/bin/pm2
argv                 : /usr/local/bin/node,/usr/local/bin/pm2,report
argv0                : node
user                 : max
uid                  : 501
gid                  : 20
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : darwin
type                 : Darwin
cpus                 : Intel(R) Core(TM) i7-7920HQ CPU @ 3.10GHz
cpus nb              : 8
freemem              : 239325184
totalmem             : 17179869184
home                 : /Users/max
===============================================================================
--- PM2 list -----------------------------------------------
┌──────────┬────┬─────────┬──────┬─────┬────────┬─────────┬────────┬─────┬─────┬──────┬──────────┐
│ App name │ id │ version │ mode │ pid │ status │ restart │ uptime │ cpu │ mem │ user │ watching │
└──────────┴────┴─────────┴──────┴─────┴────────┴─────────┴────────┴─────┴─────┴──────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/Users/max/.pm2/pm2.log last 20 lines:
PM2        | Emitted 'error' event at:
PM2        |     at ChildProcess.EventEmitter.emit (domain.js:461:12)
PM2        |     at Process.ChildProcess._handle.onexit (internal/child_process.js:246:12)
PM2        |     at onErrorNT (internal/child_process.js:415:16)
PM2        |     at process._tickCallback (internal/process/next_tick.js:63:19)
PM2        | 2020-08-02T20:43:38: PM2 log: ===============================================================================
PM2        | 2020-08-02T20:43:38: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
PM2        | 2020-08-02T20:43:38: PM2 log: Time                 : Sun Aug 02 2020 20:43:38 GMT+0300 (Eastern European Summer Time)
PM2        | 2020-08-02T20:43:38: PM2 log: PM2 version          : 3.2.9
PM2        | 2020-08-02T20:43:38: PM2 log: Node.js version      : 10.18.0
PM2        | 2020-08-02T20:43:38: PM2 log: Current arch         : x64
PM2        | 2020-08-02T20:43:38: PM2 log: PM2 home             : /Users/max/.pm2
PM2        | 2020-08-02T20:43:38: PM2 log: PM2 PID file         : /Users/max/.pm2/pm2.pid
PM2        | 2020-08-02T20:43:38: PM2 log: RPC socket file      : /Users/max/.pm2/rpc.sock
PM2        | 2020-08-02T20:43:38: PM2 log: BUS socket file      : /Users/max/.pm2/pub.sock
PM2        | 2020-08-02T20:43:38: PM2 log: Application log path : /Users/max/.pm2/logs
PM2        | 2020-08-02T20:43:38: PM2 log: Process dump file    : /Users/max/.pm2/dump.pm2
PM2        | 2020-08-02T20:43:38: PM2 log: Concurrent actions   : 2
PM2        | 2020-08-02T20:43:38: PM2 log: SIGTERM timeout      : 1600
PM2        | 2020-08-02T20:43:38: PM2 log: ===============================================================================


$ tail -n 50 ~/.pm2/pm2.log

2020-08-02T11:21:27: PM2 error: Error caught while calling pidusage
2020-08-02T11:21:27: PM2 error: Error: Error: spawn ps ENOENT
2020-08-02T11:21:57: PM2 error: Error caught while calling pidusage
2020-08-02T11:21:57: PM2 error: Error: Error: spawn ps ENOENT
2020-08-02T11:22:27: PM2 error: Error caught while calling pidusage
2020-08-02T11:22:27: PM2 error: Error: Error: spawn ps ENOENT
2020-08-02T11:22:57: PM2 error: Error caught while calling pidusage
2020-08-02T11:22:57: PM2 error: Error: Error: spawn ps ENOENT
2020-08-02T11:23:27: PM2 error: Error caught while calling pidusage
2020-08-02T11:23:27: PM2 error: Error: Error: spawn ps ENOENT
2020-08-02T11:23:57: PM2 error: [PM2][WORKER] Error caught by domain:
TypeError: Cannot read property 'on' of undefined
    at run (/usr/local/lib/node_modules/pm2/node_modules/pidusage/lib/bin.js:22:13)
    at ps (/usr/local/lib/node_modules/pm2/node_modules/pidusage/lib/ps.js:45:3)
    at get (/usr/local/lib/node_modules/pm2/node_modules/pidusage/lib/stats.js:73:3)
    at pidusage (/usr/local/lib/node_modules/pm2/node_modules/pidusage/index.js:25:5)
    at Object.getMonitorData (/usr/local/lib/node_modules/pm2/lib/God/ActionMethods.js:60:5)
    at tasks (/usr/local/lib/node_modules/pm2/lib/Worker.js:133:9)
    at Domain.<anonymous> (/usr/local/lib/node_modules/pm2/lib/Worker.js:177:7)
    at Domain.run (domain.js:349:14)
    at Timeout.wrappedTasks [as _onTimeout] (/usr/local/lib/node_modules/pm2/lib/Worker.js:176:7)
    at ontimeout (timers.js:436:11)
events.js:174
      throw er; // Unhandled 'error' event
      ^

Error: spawn ps EAGAIN
    at Process.ChildProcess._handle.onexit (internal/child_process.js:240:19)
    at onErrorNT (internal/child_process.js:415:16)
    at process._tickCallback (internal/process/next_tick.js:63:19)
Emitted 'error' event at:
    at ChildProcess.EventEmitter.emit (domain.js:461:12)
    at Process.ChildProcess._handle.onexit (internal/child_process.js:246:12)
    at onErrorNT (internal/child_process.js:415:16)
    at process._tickCallback (internal/process/next_tick.js:63:19)
2020-08-02T20:43:38: PM2 log: ===============================================================================
2020-08-02T20:43:38: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2020-08-02T20:43:38: PM2 log: Time                 : Sun Aug 02 2020 20:43:38 GMT+0300 (Eastern European Summer Time)
2020-08-02T20:43:38: PM2 log: PM2 version          : 3.2.9
2020-08-02T20:43:38: PM2 log: Node.js version      : 10.18.0
2020-08-02T20:43:38: PM2 log: Current arch         : x64
2020-08-02T20:43:38: PM2 log: PM2 home             : /Users/max/.pm2
2020-08-02T20:43:38: PM2 log: PM2 PID file         : /Users/max/.pm2/pm2.pid
2020-08-02T20:43:38: PM2 log: RPC socket file      : /Users/max/.pm2/rpc.sock
2020-08-02T20:43:38: PM2 log: BUS socket file      : /Users/max/.pm2/pub.sock
2020-08-02T20:43:38: PM2 log: Application log path : /Users/max/.pm2/logs
2020-08-02T20:43:38: PM2 log: Process dump file    : /Users/max/.pm2/dump.pm2
2020-08-02T20:43:38: PM2 log: Concurrent actions   : 2
2020-08-02T20:43:38: PM2 log: SIGTERM timeout      : 1600
2020-08-02T20:43:38: PM2 log: ===============================================================================

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied pm2 report and ~/.pm2/pm2.log, then inspect the referenced pidusage files, lib/God/ActionMethods.js, and lib/Worker.js. Investigate how the reported ps ENOENT/EAGAIN errors relate to the daemon restart and empty pm2 list. Done means the failure is reproducible or explained and the affected process-list behavior is covered by a regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
cli, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.