When disk is full, pm2 will restart continuously, causing service exceptions.

Open
#5,008 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
32/100
Issue type
Bug
Clarity
Mostly clear
Activity status
Stale
Tech stack
javascript, nodejs
Domain
cli, devops

Research direction

Reproduce the issue with the mentioned http.js example and the pm2 start command using cluster mode, merged output logs, and a full disk. Trace the cluster startup and log-writing paths involved in the repeated restarts. Done means the full-disk or permission-denied condition no longer causes continuous restarts and excessive CPU usage.

Written by the indexing model from the issue text.

Description

What's going wrong?

In cluster mode , when disk is full or permission denied to log file, pm2 will restart continuously, causing service exceptions.

How could we reproduce this issue?

1.make a small ram disk, for example 5M

diskutil erasevolume HFS+ test `hdiutil attach -nomount ram://10000`

2.write a simple nodejs project in this disk, for example http.js

var http = require('http');

var server = http.createServer(function(req, res) {
  res.writeHead(200);
  console.log('long text............')
  res.end('hey');
}).listen(process.env.PORT || 8089, '0.0.0.0', function() {
  console.log('App listening on port %d', server.address().port);
});

3.fill the disk with small files

4.then run pm2 start

pm2 start http.js  -i 8 -o out.log -e err.log --merge-logs

pm2 will restart continuously,and cpu load is very high

Supporting information

$ pm2 report
--- PM2 report ----------------------------------------------------------------
Date                 : Mon Mar 08 2021 19:53:45 GMT+0800 (GMT+08:00)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 4.5.5
node version         : 10.17.0
node path            : /usr/local/bin/pm2
argv                 : /usr/local/bin/node,/Users/yuxizhe/pm2/lib/Daemon.js
argv0                : node
user                 : yuxizhe
uid                  : 501
gid                  : 20
uptime               : 0min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 4.5.5
node version         : 10.17.0
node path            : /usr/local/bin/pm2
argv                 : /usr/local/bin/node,/usr/local/bin/pm2,report
argv0                : node
user                 : yuxizhe
uid                  : 501
gid                  : 20
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : darwin
type                 : Darwin
cpus                 : Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
cpus nb              : 4
freemem              : 1590407168
totalmem             : 17179869184
home                 : /Users/yuxizhe
===============================================================================
--- PM2 list -----------------------------------------------
┌────┬────────────────────┬──────────┬──────┬───────────┬──────────┬──────────┐
│ id │ name               │ mode     │ ↺    │ status    │ cpu      │ memory   │
├────┼────────────────────┼──────────┼──────┼───────────┼──────────┼──────────┤
│ 1  │ http               │ cluster  │ 19   │ stopped   │ 0%       │ 0b       │
│ 2  │ http               │ cluster  │ 17   │ errored   │ 0%       │ 0b       │
│ 3  │ http               │ cluster  │ 17   │ errored   │ 0%       │ 0b       │
│ 4  │ http               │ cluster  │ 18   │ online    │ 25%      │ 31.6mb   │
│ 5  │ http               │ cluster  │ 17   │ errored   │ 0%       │ 0b       │
│ 6  │ http               │ cluster  │ 17   │ online    │ 24%      │ 31.5mb   │
│ 7  │ http               │ cluster  │ 17   │ online    │ 16%      │ 27.5mb   │
│ 8  │ http               │ cluster  │ 17   │ online    │ 18%      │ 28.6mb   │
└────┴────────────────────┴──────────┴──────┴───────────┴──────────┴──────────┘
Module
┌────┬────────────────────┬──────────┬──────────┬──────────┐
│ id │ name               │ status   │ cpu      │ mem      │
├────┼────────────────────┼──────────┼──────────┼──────────┤
│ 0  │ pm2-logrotate      │ online   │ 0.3%     │ 37.2mb   │
└────┴────────────────────┴──────────┴──────────┴──────────┘
Current process list running is not in sync with saved list. Type 'pm2 save' to synchronize'
===============================================================================
--- Daemon logs --------------------------------------------
/Users/yuxizhe/.pm2/pm2.log last 20 lines:
PM2        |  2: 0x10003c063 node::AddEnvironmentCleanupHook(v8::Isolate*, void (*)(void*), void*) [/usr/local/bin/node]
PM2        |  3: 0x10010c126 node::inspector::(anonymous namespace)::AddCommandLineAPI(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]
PM2        |  4: 0x10024417f v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*) [/usr/local/bin/node]
PM2        | node /Volumes/test/http.js[80572]: ../src/inspector_js_api.cc:172:void node::inspector::(anonymous namespace)::InspectorConsoleCall(const FunctionCallbackInfo<v8::Value> &): Assertion `!inspector_method.As<Function>()->Call(context, info.Holder(), call_args.size(), call_args.data()).IsEmpty()' failed.
PM2        |  1: 0x10003d041 node::Abort() [/usr/local/bin/node]
PM2        |  2: 0x10003c063 node::AddEnvironmentCleanupHook(v8::Isolate*, void (*)(void*), void*) [/usr/local/bin/node]
PM2        |  3: 0x10010c126 node::inspector::(anonymous namespace)::AddCommandLineAPI(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]
PM2        |  4: 0x10024417f v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*) [/usr/local/bin/node]
PM2        |  5: 0x1002436c1 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/usr/local/bin/node]
PM2        |  6: 0x100242d60 v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
PM2        |  7: 0x199b520dbe3d
PM2        | 2021-03-08T19:53:45: PM2 log: App name:http id:5 disconnected
PM2        | 2021-03-08T19:53:45: PM2 log: App [http:5] exited with code [0] via signal [SIGABRT]
PM2        | 2021-03-08T19:53:45: PM2 log: Script /Volumes/test/http.js had too many unstable restarts (16). Stopped. "errored"
PM2        |  5: 0x1002436c1 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/usr/local/bin/node]
PM2        |  6: 0x100242d60 v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
PM2        |  7: 0x35e1bbedbe3d
PM2        | 2021-03-08T19:53:45: PM2 log: App name:http id:1 disconnected
PM2        | 2021-03-08T19:53:45: PM2 log: App [http:1] exited with code [0] via signal [SIGABRT]
PM2        | 2021-03-08T19:53:45: PM2 log: App [http:1] starting in -cluster mode-

--- PM2 report ----------------------------------------------------------------
Date                 : Mon Mar 08 2021 20:42:29 GMT+0800 (GMT+08:00)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 4.5.5
node version         : 10.17.0
node path            : /usr/local/bin/pm2
argv                 : /usr/local/bin/node,/Users/yuxizhe/pm2/lib/Daemon.js
argv0                : node
user                 : yuxizhe
uid                  : 501
gid                  : 20
uptime               : 0min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 4.5.5
node version         : 10.17.0
node path            : /usr/local/bin/pm2
argv                 : /usr/local/bin/node,/usr/local/bin/pm2,report
argv0                : node
user                 : yuxizhe
uid                  : 501
gid                  : 20
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : darwin
type                 : Darwin
cpus                 : Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
cpus nb              : 4
freemem              : 312184832
totalmem             : 17179869184
home                 : /Users/yuxizhe
===============================================================================
--- PM2 list -----------------------------------------------
┌────┬────────────────────┬──────────┬──────┬───────────┬──────────┬──────────┐
│ id │ name               │ mode     │ ↺    │ status    │ cpu      │ memory   │
├────┼────────────────────┼──────────┼──────┼───────────┼──────────┼──────────┤
│ 1  │ http               │ cluster  │ 9    │ online    │ 0%       │ 29.3mb   │
│ 2  │ http               │ cluster  │ 9    │ online    │ 0%       │ 24.4mb   │
│ 3  │ http               │ cluster  │ 8    │ online    │ 0%       │ 32.1mb   │
│ 4  │ http               │ cluster  │ 8    │ online    │ 0%       │ 29.4mb   │
│ 5  │ http               │ cluster  │ 8    │ online    │ 0%       │ 29.1mb   │
│ 6  │ http               │ cluster  │ 8    │ online    │ 0%       │ 24.4mb   │
│ 7  │ http               │ cluster  │ 7    │ online    │ 34%      │ 37.2mb   │
│ 8  │ http               │ cluster  │ 7    │ online    │ 0%       │ 30.5mb   │
└────┴────────────────────┴──────────┴──────┴───────────┴──────────┴──────────┘
Module
┌────┬────────────────────┬──────────┬──────────┬──────────┐
│ id │ name               │ status   │ cpu      │ mem      │
├────┼────────────────────┼──────────┼──────────┼──────────┤
│ 0  │ pm2-logrotate      │ online   │ 0.1%     │ 42.6mb   │
└────┴────────────────────┴──────────┴──────────┴──────────┘
Current process list running is not in sync with saved list. Type 'pm2 save' to synchronize'
===============================================================================
--- Daemon logs --------------------------------------------
/Users/yuxizhe/.pm2/pm2.log last 20 lines:
PM2        |  6: 0x100242d60 v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
PM2        |  7: 0x2066c94dbe3d
PM2        | 2021-03-08T20:42:29: PM2 log: App name:http id:6 disconnected
PM2        | 2021-03-08T20:42:29: PM2 log: App [http:6] exited with code [0] via signal [SIGABRT]
PM2        | 2021-03-08T20:42:29: PM2 log: App [http:6] starting in -cluster mode-
PM2        |  6: 0x100242d60 v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
PM2        |  7: 0x2cd53f1dbe3d
PM2        | 2021-03-08T20:42:29: PM2 log: App name:http id:2 disconnected
PM2        | 2021-03-08T20:42:29: PM2 log: App [http:2] exited with code [0] via signal [SIGABRT]
PM2        | 2021-03-08T20:42:29: PM2 log: App [http:2] starting in -cluster mode-
PM2        | node /Volumes/test/http.js[84705]: ../src/inspector_js_api.cc:172:void node::inspector::(anonymous namespace)::InspectorConsoleCall(const FunctionCallbackInfo<v8::Value> &): Assertion `!inspector_method.As<Function>()->Call(context, info.Holder(), call_args.size(), call_args.data()).IsEmpty()' failed.
PM2        |  1: 0x10003d041 node::Abort() [/usr/local/bin/node]
PM2        |  2: 0x10003c063 node::AddEnvironmentCleanupHook(v8::Isolate*, void (*)(void*), void*) [/usr/local/bin/node]
PM2        |  3: 0x10010c126 node::inspector::(anonymous namespace)::AddCommandLineAPI(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]
PM2        |  4: 0x10024417f v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*) [/usr/local/bin/node]
PM2        | 2021-03-08T20:42:29: PM2 log: App [http:6] online
PM2        | 2021-03-08T20:42:29: PM2 log: App [http:2] online
PM2        |  5: 0x1002436c1 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/usr/local/bin/node]
PM2        |  6: 0x100242d60 v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
PM2        |  7: 0x2754cb4dbe3d

Dominant language
JavaScript
Stars
43.3k
Forks
2.7k
PR merge metrics
No merged PRs in 30d

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from Unitech/pm2

All issues in Unitech/pm2

Similar issues

More JavaScript issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.