Unitech / Unitech/pm2

Process appear as online with wait_ready even when launch fails

Open
#5,013 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
43.3k
Forks
2.7k
PR merge metrics
No merged PRs in 30d

Description

What's going wrong?

An app started with:

"wait_ready":true,
"max_restarts":3,
"min_uptime": "2m"

Appear as online even if the app crashes at every launch and the ready signal is never emitted.

How could we reproduce this issue?

1 - Create a sample ecosystem.config.json file as

{
    "apps": [
        {
            "name": "mitown",
            "script": "index.js",
            "watch": false,
            "cwd": "someapp/",
            "exec_mode" : "cluster",
            "node_args": "--max_old_space_size=4096",
            "wait_ready":true,
            "kill_timeout": 12000,
            "listen_timeout": 100000,
            "max_memory_restart": "1536M",
            "restart_delay": 4000,
            "shutdown_with_message" : true,
            "max_restarts":3,
            "min_uptime": "2m",
            "instances":1
       }
   ]
}

2 - create someapp/index.js file with these contents:

undefinedFunctionCall()

3 - run pm2 start ecosystem.config.json

4 - wait for the app to crash 3 times, then launch pm2 list. The app appear as online with memory usage as 0b

Looks like the same issue of #4355

Supporting information

--- PM2 report ----------------------------------------------------------------
Date                 : Wed Mar 10 2021 14:33:04 GMT+0000 (Coordinated Universal Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 4.4.1
node version         : 12.20.1
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : onit
uid                  : 1001
gid                  : 1001
uptime               : 35min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 4.4.1
node version         : 12.20.1
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : onit
uid                  : 1001
gid                  : 1001
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Core(TM) i5-5300U CPU @ 2.30GHz
cpus nb              : 1
freemem              : 268554240
totalmem             : 1028964352
home                 : /home/onit
===============================================================================
--- PM2 list -----------------------------------------------
┌─────┬────────────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id  │ name           │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├─────┼────────────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 2   │ mit-ray        │ default     │ 1.0.10  │ fork    │ 32273    │ 2m     │ 0    │ online    │ 0.2%     │ 43.3mb   │ onit     │ disabled │
│ 0   │ mitown         │ default     │ 1.0.0   │ cluster │ 0        │ 2m     │ 2    │ online    │ 0%       │ 0b       │ onit     │ disabled │
│ 1   │ nats-server    │ default     │ 1.0.1   │ fork    │ 32265    │ 2m     │ 0    │ online    │ 0.2%     │ 35.7mb   │ onit     │ disabled │
└─────┴────────────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/home/onit/.pm2/pm2.log last 20 lines:
PM2        | 2021-03-10T14:30:37: PM2 log: App [nats-server:1] exited with code [0] via signal [SIGKILL]
PM2        | 2021-03-10T14:30:37: PM2 log: pid=20102 msg=process killed
PM2        | 2021-03-10T14:30:37: PM2 log: pid=20091 msg=process killed
PM2        | 2021-03-10T14:30:43: PM2 log: App [mitown:0] starting in -cluster mode-
PM2        | 2021-03-10T14:30:43: PM2 log: App [nats-server:1] starting in -fork mode-
PM2        | 2021-03-10T14:30:43: PM2 log: App [nats-server:1] online
PM2        | 2021-03-10T14:30:43: PM2 log: App [mit-ray:2] starting in -fork mode-
PM2        | 2021-03-10T14:30:43: PM2 log: App [mit-ray:2] online
PM2        | 2021-03-10T14:30:45: PM2 log: App name:mitown id:0 disconnected
PM2        | 2021-03-10T14:30:45: PM2 log: App [mitown:0] exited with code [0] via signal [SIGINT]
PM2        | 2021-03-10T14:30:49: PM2 log: App [mitown:0] starting in -cluster mode-
PM2        | 2021-03-10T14:30:49: PM2 log: App name:mitown id:0 disconnected
PM2        | 2021-03-10T14:30:49: PM2 log: App [mitown:0] exited with code [0] via signal [SIGINT]
PM2        | 2021-03-10T14:30:53: PM2 log: App [mitown:0] starting in -cluster mode-
PM2        | 2021-03-10T14:30:54: PM2 log: App name:mitown id:0 disconnected
PM2        | 2021-03-10T14:30:54: PM2 log: App [mitown:0] exited with code [0] via signal [SIGINT]
PM2        | 2021-03-10T14:30:54: PM2 log: Script /home/onit/apps/mitown/index.js had too many unstable restarts (3). Stopped. "errored"
PM2        | 2021-03-10T14:32:24: PM2 log: App [mitown:0] online
PM2        | 2021-03-10T14:32:29: PM2 log: App [mitown:0] online
PM2        | 2021-03-10T14:32:33: PM2 log: App [mitown:0] online

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the supplied ecosystem.config.json and someapp/index.js, reproduce the issue with pm2 start ecosystem.config.json, then check pm2 list and the daemon logs after three crashes. Trace why the stopped process is reported as online; done means a launch that never emits the ready signal is shown as errored or stopped rather than online.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
cli, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.