Unitech / Unitech/pm2

PM2 error: read ENOTCONN

Open
#5,338 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
43.3k
Forks
2.7k
PR merge metrics
No merged PRs in 30d

Description

What's going wrong?

It looks like pm2 while restarting an app after error getting unexpected error itself. For now I don't understand what is going and how to replicate the issue. Maybe someone can explain me an error and give me some idea of the direction where to digg.

2022-03-31T22:08:58: PM2 log: App [strategy--mm--sub1:4740] starting in -fork mode-
2022-03-31T22:08:58: PM2 log: App [strategy--mm--sub1:4740] online
2022-03-31T22:29:02: PM2 log: App [strategy--mm--sub1:4740] exited with code [1] via signal [SIGINT]
2022-03-31T22:29:12: PM2 log: App [strategy--mm--sub1:4740] starting in -fork mode-
2022-03-31T22:29:12: PM2 log: App [strategy--mm--sub1:4740] online
2022-03-31T22:29:12: PM2 error: Error: spawn /usr/bin/python3 EAGAIN
    at Process.ChildProcess._handle.onexit (node:internal/child_process:282:19)
    at onErrorNT (node:internal/child_process:477:16)
    at processTicksAndRejections (node:internal/process/task_queues:83:21)
2022-03-31T22:29:12: PM2 log: ===============================================================================
2022-03-31T22:29:12: PM2 log: --- PM2 global error caught ---------------------------------------------------
2022-03-31T22:29:12: PM2 log: Time                 : Thu Mar 31 2022 22:29:12 GMT+0000 (Coordinated Universal Time)
2022-03-31T22:29:12: PM2 error: read ENOTCONN
2022-03-31T22:29:12: PM2 error: Error: read ENOTCONN
    at tryReadStart (node:net:592:20)
    at Socket._read (node:net:603:5)
    at Socket.Readable.read (node:internal/streams/readable:487:10)
    at Socket.read (node:net:644:39)
    at new Socket (node:net:401:12)
    at Object.Socket (node:net:284:41)
    at createSocket (node:internal/child_process:327:14)
    at ChildProcess.spawn (node:internal/child_process:444:23)
    at spawn (node:child_process:698:9)
    at /usr/lib/node_modules/pm2/lib/God/ForkMode.js:118:20
2022-03-31T22:29:12: PM2 log: ===============================================================================
2022-03-31T22:29:12: PM2 error: [PM2] Resurrecting PM2
node:events:368
      throw er; // Unhandled 'error' event
      ^

<ref *1> Error: read ENOTCONN
    at tryReadStart (node:net:592:20)
    at Socket._read (node:net:603:5)
    at Socket.Readable.read (node:internal/streams/readable:487:10)
    at Socket.read (node:net:644:39)
    at new Socket (node:net:401:12)
    at Object.Socket (node:net:284:41)
    at createSocket (node:internal/child_process:327:14)
    at ChildProcess.spawn (node:internal/child_process:444:23)
    at spawn (node:child_process:698:9)
    at /usr/lib/node_modules/pm2/lib/God/ForkMode.js:118:20
Emitted 'error' event on Domain instance at:
    at Socket.emit (node:domain:525:12)
    at emitErrorNT (node:internal/streams/destroy:157:8)
    at emitErrorCloseNT (node:internal/streams/destroy:122:3)
    at processTicksAndRejections (node:internal/process/task_queues:83:21) {
  errno: -107,
  code: 'ENOTCONN',
  syscall: 'read',
  domainEmitter: Socket {
    connecting: false,
    _hadError: false,
    _parent: null,
    _host: null,
    _readableState: ReadableState {
      objectMode: false,
      highWaterMark: 16384,
      buffer: BufferList { head: null, tail: null, length: 0 },
      length: 0,
      pipes: [],
      flowing: true,
      ended: false,
      endEmitted: false,
      reading: true,
      constructed: true,
      sync: false,
      needReadable: true,
      emittedReadable: false,
      readableListening: false,
      resumeScheduled: true,
      errorEmitted: true,
      emitClose: false,
      autoDestroy: true,
      destroyed: true,
      errored: [Circular *1],
      closed: true,
      closeEmitted: false,
      defaultEncoding: 'utf8',
      awaitDrainWriters: null,
      multiAwaitDrain: false,
      readingMore: false,
      dataEmitted: false,
      decoder: null,
      encoding: null,
      [Symbol(kPaused)]: false
    },
    _events: [Object: null prototype] {
      end: [Function: onReadableStreamEnd],
      close: [Function (anonymous)],
      data: [Function: forkErrData]
    },
    _eventsCount: 3,
    _maxListeners: undefined,
    _writableState: WritableState {
      objectMode: false,
      highWaterMark: 16384,
      finalCalled: false,
      needDrain: false,
      ending: false,
      ended: false,
      finished: false,
      destroyed: true,
      decodeStrings: false,
      defaultEncoding: 'utf8',
      length: 0,
      writing: false,
      corked: 0,
      sync: true,
      bufferProcessing: false,
      onwrite: [Function: bound onwrite],
      writecb: null,
      writelen: 0,
      afterWriteTickInfo: null,
      buffered: [],
      bufferedIndex: 0,
      allBuffers: true,
      allNoop: true,
      pendingcb: 0,
      constructed: true,
      prefinished: false,
      errorEmitted: true,
      emitClose: false,
      autoDestroy: true,
      errored: [Circular *1],
      closed: true,
      closeEmitted: false,
      writable: false,
      [Symbol(kOnFinished)]: []
    },
    allowHalfOpen: false,
    _sockname: null,
    _pendingData: null,
    _pendingEncoding: '',
    server: null,
    _server: null,
    [Symbol(async_id_symbol)]: 92180833,
    [Symbol(kHandle)]: null,
    [Symbol(kSetNoDelay)]: false,
    [Symbol(lastWriteQueueSize)]: 0,
    [Symbol(timeout)]: null,
    [Symbol(kBuffer)]: null,
    [Symbol(kBufferCb)]: null,
    [Symbol(kBufferGen)]: null,
    [Symbol(kCapture)]: false,
    [Symbol(kBytesRead)]: 0,
    [Symbol(kBytesWritten)]: 0
  },
  domainThrown: false
}

How could we reproduce this issue?

Right now have no idea.

Supporting information

--- PM2 report ----------------------------------------------------------------
Date                 : Fri Apr 01 2022 11:20:44 GMT+0000 (Coordinated Universal Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 5.1.2
node version         : 16.14.2
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : ec2-user
uid                  : 1000
gid                  : 1000
uptime               : 3min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 5.1.2
node version         : 16.14.2
node path            : /usr/bin/pm2
argv                 : /usr/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : ec2-user
uid                  : 1000
gid                  : 1000
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Xeon(R) Platinum 8124M CPU @ 3.00GHz
cpus nb              : 36
freemem              : 30082981888
totalmem             : 73638830080
home                 : /home/ec2-user
===============================================================================

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

The stack trace points to PM2's lib/God/ForkMode.js and Node.js child_process.spawn; start by tracing the restart path around the reported EAGAIN and ENOTCONN errors. Try to reproduce it with PM2 5.1.2, Node.js 16.14.2, and Linux; done means the failure has a reproducible cause and the daemon's restart behavior is verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, linux, node.js
Domain
cli, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.