Unitech / Unitech/pm2

PM2 daemon crashing during process restart

Open
#1,320 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

S: Icebox T: Bug
Dominant language
JavaScript
Stars
43.3k
Forks
2.7k
PR merge metrics
No merged PRs in 30d

Description

PM2 is regularly crashing for me when restarting (or reloading) a process. This also occurs with the startOr_______ versions as well. However, sometimes the restart will work smoothly, so it's not happening 100% of the time.

Relevant versions:

  • pm2: 0.12.15
  • Node (io.js): v2.0.2
  • Ubuntu: 14.04.2 LTS (I can't reproduce this in OS X)

Below, you'll see it in action. The steps I take are:

  1. pm2 l to show what's already running
  2. pm2 startOrRestart ecosystem.json to trigger a restart
  3. Everything stops after [PM2] restartProcessId process id 0 is printed, and I need to cancel with ^C; if I were to wait instead, it would be stuck at this step indefinitely
  4. pm2 l again to see what is running, which appears to spawn a new PM2 daemon
  5. pm2 start ecosystem.json to get api-server running again
$ pm2 l
┌────────────┬────┬─────────┬───────┬────────┬─────────┬────────┬──────────────┬──────────┐
│ App name   │ id │ mode    │ pid   │ status │ restart │ uptime │ memory       │ watching │
├────────────┼────┼─────────┼───────┼────────┼─────────┼────────┼──────────────┼──────────┤
│ api-server │ 0  │ cluster │ 22164 │ online │ 1       │ 33s    │ 264.336 MB   │ disabled │
│ api-server │ 1  │ cluster │ 22190 │ online │ 1       │ 33s    │ 375.887 MB   │ disabled │
└────────────┴────┴─────────┴───────┴────────┴─────────┴────────┴──────────────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app

$ pm2 startOrRestart ecosystem.json
[PM2] restartProcessId process id 0
^C

$ pm2 l
[PM2] Spawning PM2 daemon
[PM2] PM2 Successfully daemonized
┌──────────┬────┬──────┬─────┬────────┬─────────┬────────┬────────┬──────────┐
│ App name │ id │ mode │ pid │ status │ restart │ uptime │ memory │ watching │
└──────────┴────┴──────┴─────┴────────┴─────────┴────────┴────────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app

$ pm2 start ecosystem.json
[PM2] Process launched
┌────────────┬────┬─────────┬───────┬────────┬─────────┬────────┬─────────────┬──────────┐
│ App name   │ id │ mode    │ pid   │ status │ restart │ uptime │ memory      │ watching │
├────────────┼────┼─────────┼───────┼────────┼─────────┼────────┼─────────────┼──────────┤
│ api-server │ 0  │ cluster │ 22398 │ online │ 0       │ 0s     │ 28.734 MB   │ disabled │
│ api-server │ 1  │ cluster │ 22408 │ online │ 0       │ 0s     │ 16.570 MB   │ disabled │
└────────────┴────┴─────────┴───────┴────────┴─────────┴────────┴─────────────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app

We're using keymetrics, but I removed some ● Agent online lines from the output above.

pm2.log during that time:

2015-05-29 13:36:47: Stopping app:api-server id:0
2015-05-29 13:36:47: App name:api-server id:0 disconnected
2015-05-29 13:36:47: App name:api-server id:0 exited with code SIGTERM
2015-05-29 13:36:47: Process with pid 22164 killed
2015-05-29 13:36:47: Starting execution sequence in -cluster mode- for app name:api-server id:0
2015-05-29 13:36:48: App name:api-server id:0 online
2015-05-29 13:36:48: Stopping app:api-server id:1
2015-05-29 13:36:48: App name:api-server id:1 disconnected
2015-05-29 13:36:48: App name:api-server id:1 exited with code 100
2015-05-29 13:37:04: [PM2][WORKER] Started with refreshing interval: 30000
2015-05-29 13:37:04: [[[[ PM2/God daemon launched ]]]]
2015-05-29 13:37:04: BUS system [READY] on port /home/ubuntu/.pm2/pub.sock
2015-05-29 13:37:04: RPC interface [READY] on port /home/ubuntu/.pm2/rpc.sock
2015-05-29 13:37:11: Starting execution sequence in -cluster mode- for app name:api-server id:0
2015-05-29 13:37:11: App name:api-server id:0 online
2015-05-29 13:37:11: Starting execution sequence in -cluster mode- for app name:api-server id:1
2015-05-29 13:37:11: App name:api-server id:1 online

ecosystem.json:

{
  "apps" : [
    {
      "name": "api-server",
      "script": "app.js",
      "exec_mode": "cluster",
      "instances": 0,
      "env": {
        "NODE_ENV": "staging"
      }
    }
  ]
}

Please let me know if there is any additional information you need, or if I'm doing something incorrectly. Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing pm2 startOrRestart ecosystem.json with the listed PM2, io.js, and Ubuntu versions, then compare the command output with pm2.log. Use the provided ecosystem.json and cluster-mode restart sequence as the test case. Done means restarting does not hang or require interruption, and the PM2 daemon remains available without spawning a replacement.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
cli, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.