When disk is full, pm2 will restart continuously, causing service exceptions.
Nobody has claimed this yet.
Assessment
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Newbie friendliness
- 32/100
- Issue type
- Bug
- Clarity
- Mostly clear
- Activity status
- Stale
- Tech stack
- javascript, nodejs
Research direction
Reproduce the issue with the mentioned http.js example and the pm2 start command using cluster mode, merged output logs, and a full disk. Trace the cluster startup and log-writing paths involved in the repeated restarts. Done means the full-disk or permission-denied condition no longer causes continuous restarts and excessive CPU usage.
Written by the indexing model from the issue text.
Description
What's going wrong?
In cluster mode , when disk is full or permission denied to log file, pm2 will restart continuously, causing service exceptions.
How could we reproduce this issue?
1.make a small ram disk, for example 5M
diskutil erasevolume HFS+ test `hdiutil attach -nomount ram://10000`
2.write a simple nodejs project in this disk, for example http.js
var http = require('http');
var server = http.createServer(function(req, res) {
res.writeHead(200);
console.log('long text............')
res.end('hey');
}).listen(process.env.PORT || 8089, '0.0.0.0', function() {
console.log('App listening on port %d', server.address().port);
});
3.fill the disk with small files
4.then run pm2 start
pm2 start http.js -i 8 -o out.log -e err.log --merge-logs
pm2 will restart continuously,and cpu load is very high
Supporting information
$ pm2 report
--- PM2 report ----------------------------------------------------------------
Date : Mon Mar 08 2021 19:53:45 GMT+0800 (GMT+08:00)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version : 4.5.5
node version : 10.17.0
node path : /usr/local/bin/pm2
argv : /usr/local/bin/node,/Users/yuxizhe/pm2/lib/Daemon.js
argv0 : node
user : yuxizhe
uid : 501
gid : 20
uptime : 0min
===============================================================================
--- CLI ----------------------------------------------------
local pm2 : 4.5.5
node version : 10.17.0
node path : /usr/local/bin/pm2
argv : /usr/local/bin/node,/usr/local/bin/pm2,report
argv0 : node
user : yuxizhe
uid : 501
gid : 20
===============================================================================
--- System info --------------------------------------------
arch : x64
platform : darwin
type : Darwin
cpus : Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
cpus nb : 4
freemem : 1590407168
totalmem : 17179869184
home : /Users/yuxizhe
===============================================================================
--- PM2 list -----------------------------------------------
┌────┬────────────────────┬──────────┬──────┬───────────┬──────────┬──────────┐
│ id │ name │ mode │ ↺ │ status │ cpu │ memory │
├────┼────────────────────┼──────────┼──────┼───────────┼──────────┼──────────┤
│ 1 │ http │ cluster │ 19 │ stopped │ 0% │ 0b │
│ 2 │ http │ cluster │ 17 │ errored │ 0% │ 0b │
│ 3 │ http │ cluster │ 17 │ errored │ 0% │ 0b │
│ 4 │ http │ cluster │ 18 │ online │ 25% │ 31.6mb │
│ 5 │ http │ cluster │ 17 │ errored │ 0% │ 0b │
│ 6 │ http │ cluster │ 17 │ online │ 24% │ 31.5mb │
│ 7 │ http │ cluster │ 17 │ online │ 16% │ 27.5mb │
│ 8 │ http │ cluster │ 17 │ online │ 18% │ 28.6mb │
└────┴────────────────────┴──────────┴──────┴───────────┴──────────┴──────────┘
Module
┌────┬────────────────────┬──────────┬──────────┬──────────┐
│ id │ name │ status │ cpu │ mem │
├────┼────────────────────┼──────────┼──────────┼──────────┤
│ 0 │ pm2-logrotate │ online │ 0.3% │ 37.2mb │
└────┴────────────────────┴──────────┴──────────┴──────────┘
Current process list running is not in sync with saved list. Type 'pm2 save' to synchronize'
===============================================================================
--- Daemon logs --------------------------------------------
/Users/yuxizhe/.pm2/pm2.log last 20 lines:
PM2 | 2: 0x10003c063 node::AddEnvironmentCleanupHook(v8::Isolate*, void (*)(void*), void*) [/usr/local/bin/node]
PM2 | 3: 0x10010c126 node::inspector::(anonymous namespace)::AddCommandLineAPI(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]
PM2 | 4: 0x10024417f v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*) [/usr/local/bin/node]
PM2 | node /Volumes/test/http.js[80572]: ../src/inspector_js_api.cc:172:void node::inspector::(anonymous namespace)::InspectorConsoleCall(const FunctionCallbackInfo<v8::Value> &): Assertion `!inspector_method.As<Function>()->Call(context, info.Holder(), call_args.size(), call_args.data()).IsEmpty()' failed.
PM2 | 1: 0x10003d041 node::Abort() [/usr/local/bin/node]
PM2 | 2: 0x10003c063 node::AddEnvironmentCleanupHook(v8::Isolate*, void (*)(void*), void*) [/usr/local/bin/node]
PM2 | 3: 0x10010c126 node::inspector::(anonymous namespace)::AddCommandLineAPI(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]
PM2 | 4: 0x10024417f v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*) [/usr/local/bin/node]
PM2 | 5: 0x1002436c1 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/usr/local/bin/node]
PM2 | 6: 0x100242d60 v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
PM2 | 7: 0x199b520dbe3d
PM2 | 2021-03-08T19:53:45: PM2 log: App name:http id:5 disconnected
PM2 | 2021-03-08T19:53:45: PM2 log: App [http:5] exited with code [0] via signal [SIGABRT]
PM2 | 2021-03-08T19:53:45: PM2 log: Script /Volumes/test/http.js had too many unstable restarts (16). Stopped. "errored"
PM2 | 5: 0x1002436c1 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/usr/local/bin/node]
PM2 | 6: 0x100242d60 v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
PM2 | 7: 0x35e1bbedbe3d
PM2 | 2021-03-08T19:53:45: PM2 log: App name:http id:1 disconnected
PM2 | 2021-03-08T19:53:45: PM2 log: App [http:1] exited with code [0] via signal [SIGABRT]
PM2 | 2021-03-08T19:53:45: PM2 log: App [http:1] starting in -cluster mode-
--- PM2 report ----------------------------------------------------------------
Date : Mon Mar 08 2021 20:42:29 GMT+0800 (GMT+08:00)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version : 4.5.5
node version : 10.17.0
node path : /usr/local/bin/pm2
argv : /usr/local/bin/node,/Users/yuxizhe/pm2/lib/Daemon.js
argv0 : node
user : yuxizhe
uid : 501
gid : 20
uptime : 0min
===============================================================================
--- CLI ----------------------------------------------------
local pm2 : 4.5.5
node version : 10.17.0
node path : /usr/local/bin/pm2
argv : /usr/local/bin/node,/usr/local/bin/pm2,report
argv0 : node
user : yuxizhe
uid : 501
gid : 20
===============================================================================
--- System info --------------------------------------------
arch : x64
platform : darwin
type : Darwin
cpus : Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz
cpus nb : 4
freemem : 312184832
totalmem : 17179869184
home : /Users/yuxizhe
===============================================================================
--- PM2 list -----------------------------------------------
┌────┬────────────────────┬──────────┬──────┬───────────┬──────────┬──────────┐
│ id │ name │ mode │ ↺ │ status │ cpu │ memory │
├────┼────────────────────┼──────────┼──────┼───────────┼──────────┼──────────┤
│ 1 │ http │ cluster │ 9 │ online │ 0% │ 29.3mb │
│ 2 │ http │ cluster │ 9 │ online │ 0% │ 24.4mb │
│ 3 │ http │ cluster │ 8 │ online │ 0% │ 32.1mb │
│ 4 │ http │ cluster │ 8 │ online │ 0% │ 29.4mb │
│ 5 │ http │ cluster │ 8 │ online │ 0% │ 29.1mb │
│ 6 │ http │ cluster │ 8 │ online │ 0% │ 24.4mb │
│ 7 │ http │ cluster │ 7 │ online │ 34% │ 37.2mb │
│ 8 │ http │ cluster │ 7 │ online │ 0% │ 30.5mb │
└────┴────────────────────┴──────────┴──────┴───────────┴──────────┴──────────┘
Module
┌────┬────────────────────┬──────────┬──────────┬──────────┐
│ id │ name │ status │ cpu │ mem │
├────┼────────────────────┼──────────┼──────────┼──────────┤
│ 0 │ pm2-logrotate │ online │ 0.1% │ 42.6mb │
└────┴────────────────────┴──────────┴──────────┴──────────┘
Current process list running is not in sync with saved list. Type 'pm2 save' to synchronize'
===============================================================================
--- Daemon logs --------------------------------------------
/Users/yuxizhe/.pm2/pm2.log last 20 lines:
PM2 | 6: 0x100242d60 v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
PM2 | 7: 0x2066c94dbe3d
PM2 | 2021-03-08T20:42:29: PM2 log: App name:http id:6 disconnected
PM2 | 2021-03-08T20:42:29: PM2 log: App [http:6] exited with code [0] via signal [SIGABRT]
PM2 | 2021-03-08T20:42:29: PM2 log: App [http:6] starting in -cluster mode-
PM2 | 6: 0x100242d60 v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
PM2 | 7: 0x2cd53f1dbe3d
PM2 | 2021-03-08T20:42:29: PM2 log: App name:http id:2 disconnected
PM2 | 2021-03-08T20:42:29: PM2 log: App [http:2] exited with code [0] via signal [SIGABRT]
PM2 | 2021-03-08T20:42:29: PM2 log: App [http:2] starting in -cluster mode-
PM2 | node /Volumes/test/http.js[84705]: ../src/inspector_js_api.cc:172:void node::inspector::(anonymous namespace)::InspectorConsoleCall(const FunctionCallbackInfo<v8::Value> &): Assertion `!inspector_method.As<Function>()->Call(context, info.Holder(), call_args.size(), call_args.data()).IsEmpty()' failed.
PM2 | 1: 0x10003d041 node::Abort() [/usr/local/bin/node]
PM2 | 2: 0x10003c063 node::AddEnvironmentCleanupHook(v8::Isolate*, void (*)(void*), void*) [/usr/local/bin/node]
PM2 | 3: 0x10010c126 node::inspector::(anonymous namespace)::AddCommandLineAPI(v8::FunctionCallbackInfo<v8::Value> const&) [/usr/local/bin/node]
PM2 | 4: 0x10024417f v8::internal::FunctionCallbackArguments::Call(v8::internal::CallHandlerInfo*) [/usr/local/bin/node]
PM2 | 2021-03-08T20:42:29: PM2 log: App [http:6] online
PM2 | 2021-03-08T20:42:29: PM2 log: App [http:2] online
PM2 | 5: 0x1002436c1 v8::internal::MaybeHandle<v8::internal::Object> v8::internal::(anonymous namespace)::HandleApiCallHelper<false>(v8::internal::Isolate*, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::HeapObject>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>, v8::internal::Handle<v8::internal::Object>, v8::internal::BuiltinArguments) [/usr/local/bin/node]
PM2 | 6: 0x100242d60 v8::internal::Builtin_Impl_HandleApiCall(v8::internal::BuiltinArguments, v8::internal::Isolate*) [/usr/local/bin/node]
PM2 | 7: 0x2754cb4dbe3d
- Dominant language
- JavaScript
- Stars
- 43.3k
- Forks
- 2.7k
- PR merge metrics
- No merged PRs in 30d
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
More from Unitech/pm2
-
Difficulty 2/5 1-3 hours Newbie friendliness 72/100
-
Difficulty 1/5 Under an hour Newbie friendliness 90/100
-
Difficulty 3/5 1-2 days Newbie friendliness 72/100
-
Errors Open
Difficulty 4/5 3-5 days Newbie friendliness 28/100
-
Difficulty 4/5 3-5 days Newbie friendliness 35/100
Similar issues
-
code-quality refactoring
Difficulty 2/5 1-3 hours Newbie friendliness 84/100
github/gh-aw-firewall#8816 ·
-
integration:quickjs org:external priority:backlog topic:code-interpreter topic:middleware type:feature
Difficulty 2/5 1-3 hours Newbie friendliness 74/100
langchain-ai/deepagents#6450 ·
-
optimization optimization:agents-md-curator
Difficulty 2/5 1-3 hours Newbie friendliness 86/100
githubnext/gh-aw-cao#13143 ·
-
status: needs triage
Difficulty 2/5 1-3 hours Newbie friendliness 88/100
-
Difficulty 2/5 1-3 hours Newbie friendliness 88/100