PM2 error: read ENOTCONN
Open
Nobody has claimed this yet.
- Dominant language
- JavaScript
- Stars
- 43.3k
- Forks
- 2.7k
- PR merge metrics
- No merged PRs in 30d
Description
What's going wrong?
It looks like pm2 while restarting an app after error getting unexpected error itself. For now I don't understand what is going and how to replicate the issue. Maybe someone can explain me an error and give me some idea of the direction where to digg.
2022-03-31T22:08:58: PM2 log: App [strategy--mm--sub1:4740] starting in -fork mode-
2022-03-31T22:08:58: PM2 log: App [strategy--mm--sub1:4740] online
2022-03-31T22:29:02: PM2 log: App [strategy--mm--sub1:4740] exited with code [1] via signal [SIGINT]
2022-03-31T22:29:12: PM2 log: App [strategy--mm--sub1:4740] starting in -fork mode-
2022-03-31T22:29:12: PM2 log: App [strategy--mm--sub1:4740] online
2022-03-31T22:29:12: PM2 error: Error: spawn /usr/bin/python3 EAGAIN
at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19)
at onErrorNT (node:internal/child_process:477:16)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
2022-03-31T22:29:12: PM2 log: ===============================================================================
2022-03-31T22:29:12: PM2 log: --- PM2 global error caught ---------------------------------------------------
2022-03-31T22:29:12: PM2 log: Time : Thu Mar 31 2022 22:29:12 GMT+0000 (Coordinated Universal Time)
2022-03-31T22:29:12: PM2 error: read ENOTCONN
2022-03-31T22:29:12: PM2 error: Error: read ENOTCONN
at tryReadStart (node:net:592:20)
at Socket._read (node:net:603:5)
at Socket.Readable.read (node:internal/streams/readable:487:10)
at Socket.read (node:net:644:39)
at new Socket (node:net:401:12)
at Object.Socket (node:net:284:41)
at createSocket (node:internal/child_process:327:14)
at ChildProcess.spawn (node:internal/child_process:444:23)
at spawn (node:child_process:698:9)
at /usr/lib/node_modules/pm2/lib/God/ForkMode.js:118:20
2022-03-31T22:29:12: PM2 log: ===============================================================================
2022-03-31T22:29:12: PM2 error: [PM2] Resurrecting PM2
node:events:368
throw er; // Unhandled 'error' event
^
<ref *1> Error: read ENOTCONN
at tryReadStart (node:net:592:20)
at Socket._read (node:net:603:5)
at Socket.Readable.read (node:internal/streams/readable:487:10)
at Socket.read (node:net:644:39)
at new Socket (node:net:401:12)
at Object.Socket (node:net:284:41)
at createSocket (node:internal/child_process:327:14)
at ChildProcess.spawn (node:internal/child_process:444:23)
at spawn (node:child_process:698:9)
at /usr/lib/node_modules/pm2/lib/God/ForkMode.js:118:20
Emitted 'error' event on Domain instance at:
at Socket.emit (node:domain:525:12)
at emitErrorNT (node:internal/streams/destroy:157:8)
at emitErrorCloseNT (node:internal/streams/destroy:122:3)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
errno: -107,
code: 'ENOTCONN',
syscall: 'read',
domainEmitter: Socket {
connecting: false,
_hadError: false,
_parent: null,
_host: null,
_readableState: ReadableState {
objectMode: false,
highWaterMark: 16384,
buffer: BufferList { head: null, tail: null, length: 0 },
length: 0,
pipes: [],
flowing: true,
ended: false,
endEmitted: false,
reading: true,
constructed: true,
sync: false,
needReadable: true,
emittedReadable: false,
readableListening: false,
resumeScheduled: true,
errorEmitted: true,
emitClose: false,
autoDestroy: true,
destroyed: true,
errored: [Circular *1],
closed: true,
closeEmitted: false,
defaultEncoding: 'utf8',
awaitDrainWriters: null,
multiAwaitDrain: false,
readingMore: false,
dataEmitted: false,
decoder: null,
encoding: null,
[Symbol(kPaused)]: false
},
_events: [Object: null prototype] {
end: [Function: onReadableStreamEnd],
close: [Function (anonymous)],
data: [Function: forkErrData]
},
_eventsCount: 3,
_maxListeners: undefined,
_writableState: WritableState {
objectMode: false,
highWaterMark: 16384,
finalCalled: false,
needDrain: false,
ending: false,
ended: false,
finished: false,
destroyed: true,
decodeStrings: false,
defaultEncoding: 'utf8',
length: 0,
writing: false,
corked: 0,
sync: true,
bufferProcessing: false,
onwrite: [Function: bound onwrite],
writecb: null,
writelen: 0,
afterWriteTickInfo: null,
buffered: [],
bufferedIndex: 0,
allBuffers: true,
allNoop: true,
pendingcb: 0,
constructed: true,
prefinished: false,
errorEmitted: true,
emitClose: false,
autoDestroy: true,
errored: [Circular *1],
closed: true,
closeEmitted: false,
writable: false,
[Symbol(kOnFinished)]: []
},
allowHalfOpen: false,
_sockname: null,
_pendingData: null,
_pendingEncoding: '',
server: null,
_server: null,
[Symbol(async_id_symbol)]: 92180833,
[Symbol(kHandle)]: null,
[Symbol(kSetNoDelay)]: false,
[Symbol(lastWriteQueueSize)]: 0,
[Symbol(timeout)]: null,
[Symbol(kBuffer)]: null,
[Symbol(kBufferCb)]: null,
[Symbol(kBufferGen)]: null,
[Symbol(kCapture)]: false,
[Symbol(kBytesRead)]: 0,
[Symbol(kBytesWritten)]: 0
},
domainThrown: false
}
How could we reproduce this issue?
Right now have no idea.
Supporting information
--- PM2 report ----------------------------------------------------------------
Date : Fri Apr 01 2022 11:20:44 GMT+0000 (Coordinated Universal Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version : 5.1.2
node version : 16.14.2
node path : /usr/bin/pm2
argv : /usr/bin/node,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0 : node
user : ec2-user
uid : 1000
gid : 1000
uptime : 3min
===============================================================================
--- CLI ----------------------------------------------------
local pm2 : 5.1.2
node version : 16.14.2
node path : /usr/bin/pm2
argv : /usr/bin/node,/usr/bin/pm2,report
argv0 : node
user : ec2-user
uid : 1000
gid : 1000
===============================================================================
--- System info --------------------------------------------
arch : x64
platform : linux
type : Linux
cpus : Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
cpus nb : 36
freemem : 30082981888
totalmem : 73638830080
home : /home/ec2-user
===============================================================================
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The stack trace points to PM2's lib/God/ForkMode.js and Node.js child_process.spawn; start by tracing the restart path around the reported EAGAIN and ENOTCONN errors. Try to reproduce it with PM2 5.1.2, Node.js 16.14.2, and Linux; done means the failure has a reproducible cause and the daemon's restart behavior is verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, linux, node.js
- Domain
- cli, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100