Unitech / Unitech/pm2

PM2 stopped cause of Error: write EPIPE error

Open
#5,538 4 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
43.3k
Forks
2.7k
No merged PRs in 30d

Description

What's going wrong?

Last night our pm2 randomly stopped running after two weeks without any problems.
After running pm2 resurrect the process restarted without any issues.
When searching through the logs i found the following errors:

2023-01-25T20:00:17: PM2 log: App [app.compiled:13] starting in -cluster mode-
2023-01-25T20:00:17: PM2 log: App [app.compiled:10] starting in -cluster mode-
2023-01-25T20:00:17: PM2 log: App [app.compiled:1] starting in -cluster mode-
2023-01-25T20:00:17: PM2 error: Error: write EPIPE
    at ChildProcess.target._send (node:internal/child_process:874:20)
    at ChildProcess.<anonymous> (node:internal/child_process:677:16)
    at ChildProcess.emit (node:events:525:35)
    at ChildProcess.emit (node:domain:552:15)
    at emit (node:internal/child_process:946:14)
    at processTicksAndRejections (node:internal/process/task_queues:84:21)
2023-01-25T20:00:17: PM2 log: ===============================================================================
2023-01-25T20:00:17: PM2 log: --- PM2 global error caught ---------------------------------------------------
2023-01-25T20:00:17: PM2 log: Time                 : Wed Jan 25 2023 20:00:17 GMT+0000 (Coordinated Universal Time)
2023-01-25T20:00:17: PM2 error: write EPIPE
2023-01-25T20:00:17: PM2 error: Error: write EPIPE
    at ChildProcess.target._send (node:internal/child_process:874:20)
    at ChildProcess.target.send (node:internal/child_process:747:19)
    at sendHelper (node:internal/cluster/utils:28:15)
    at send (node:internal/cluster/primary:363:10)
    at Worker.disconnect (node:internal/cluster/primary:369:3)
    at Worker.destroy (node:internal/cluster/primary:382:10)
    at Worker.<anonymous> (/usr/lib/node_modules/pm2/lib/God.js:248:30)
    at Object.onceWrapper (node:events:628:26)
    at Worker.emit (node:events:513:28)
    at Worker.emit (node:domain:489:12)
2023-01-25T20:00:17: PM2 log: ===============================================================================
2023-01-25T20:00:17: PM2 error: [PM2] Resurrecting PM2
2023-01-25T20:00:17: PM2 log: App name:app.compiled id:24 disconnected
2023-01-25T20:00:17: PM2 log: App [app.compiled:32] exited with code [0] via signal [SIGINT]
2023-01-25T20:00:17: PM2 log: App [app.compiled:24] exited with code [0] via signal [SIGINT]
2023-01-25T20:00:17: PM2 log: App [app.compiled:32] starting in -cluster mode-
2023-01-25T20:00:17: PM2 log: App [app.compiled:24] starting in -cluster mode-
2023-01-25T20:00:17: PM2 log: App [app.compiled:13] online
2023-01-25T20:00:17: PM2 log: App [app.compiled:1] online
node:internal/modules/cjs/loader:988
  throw err;
  ^

Error: Cannot find module '/undefined'
    at Function.Module._resolveFilename (node:internal/modules/cjs/loader:985:15)
    at Function.Module._load (node:internal/modules/cjs/loader:833:27)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12)
    at node:internal/main/run_main_module:22:47 {
  code: 'MODULE_NOT_FOUND',
  requireStack: []
}
2023-01-25T20:00:17: PM2 log: App [app.compiled:32] online
2023-01-25T20:00:17: PM2 log: App [app.compiled:10] online
2023-01-25T20:00:17: PM2 log: PM2 successfully forked
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at process.target._send (node:internal/child_process:874:20)
    at process.target.send (node:internal/child_process:747:19)
    at IPCTransport.send (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:83:21)
    at IPCTransport.setOptions (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:73:21)
    at Function.configureModule (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/configuration.js:13:62)
    at NotifyFeature.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/features/notify.js:33:33)
    at FeatureManager.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/featureManager.js:81:21)
    at PMX.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/pmx.js:90:29)
    at Object.<anonymous> (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/index.js:5:65)
    at Module._compile (node:internal/modules/cjs/loader:1155:14)
Emitted 'error' event on Worker instance at:
    at process.<anonymous> (node:internal/cluster/worker:32:12)
    at process.emit (node:events:513:28)
    at node:internal/child_process:878:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at process.target._send (node:internal/child_process:874:20)
    at process.target.send (node:internal/child_process:747:19)
    at IPCTransport.send (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:83:21)
    at IPCTransport.setOptions (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:73:21)
    at Function.configureModule (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/configuration.js:13:62)
    at NotifyFeature.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/features/notify.js:33:33)
    at FeatureManager.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/featureManager.js:81:21)
    at PMX.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/pmx.js:90:29)
    at Object.<anonymous> (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/index.js:5:65)
    at Module._compile (node:internal/modules/cjs/loader:1155:14)
Emitted 'error' event on Worker instance at:
    at process.<anonymous> (node:internal/cluster/worker:32:12)
    at process.emit (node:events:513:28)
    at node:internal/child_process:878:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at process.target._send (node:internal/child_process:874:20)
    at process.target.send (node:internal/child_process:747:19)
    at IPCTransport.send (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:83:21)
    at IPCTransport.setOptions (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:73:21)
    at Function.configureModule (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/configuration.js:13:62)
    at NotifyFeature.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/features/notify.js:33:33)
    at FeatureManager.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/featureManager.js:81:21)
    at PMX.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/pmx.js:90:29)
    at Object.<anonymous> (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/index.js:5:65)
    at Module._compile (node:internal/modules/cjs/loader:1155:14)
Emitted 'error' event on Worker instance at:
    at process.<anonymous> (node:internal/cluster/worker:32:12)
    at process.emit (node:events:513:28)
    at node:internal/child_process:878:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at process.target._send (node:internal/child_process:874:20)
    at process.target.send (node:internal/child_process:747:19)
    at IPCTransport.send (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:83:21)
    at IPCTransport.setOptions (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:73:21)
    at Function.configureModule (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/configuration.js:13:62)
    at NotifyFeature.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/features/notify.js:33:33)
    at FeatureManager.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/featureManager.js:81:21)
    at PMX.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/pmx.js:90:29)
    at Object.<anonymous> (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/index.js:5:65)
    at Module._compile (node:internal/modules/cjs/loader:1155:14)
Emitted 'error' event on Worker instance at:
    at process.<anonymous> (node:internal/cluster/worker:32:12)
    at process.emit (node:events:513:28)
    at node:internal/child_process:878:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}
node:events:491
      throw er; // Unhandled 'error' event
      ^

Error: write EPIPE
    at process.target._send (node:internal/child_process:874:20)
    at process.target.send (node:internal/child_process:747:19)
    at IPCTransport.send (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:83:21)
    at IPCTransport.setOptions (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/transports/IPCTransport.js:73:21)
    at Function.configureModule (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/configuration.js:13:62)
    at NotifyFeature.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/features/notify.js:33:33)
    at FeatureManager.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/featureManager.js:81:21)
    at PMX.init (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/pmx.js:90:29)
    at Object.<anonymous> (/usr/lib/node_modules/pm2/node_modules/@pm2/io/build/main/index.js:5:65)
    at Module._compile (node:internal/modules/cjs/loader:1155:14)
Emitted 'error' event on Worker instance at:
    at process.<anonymous> (node:internal/cluster/worker:32:12)
    at process.emit (node:events:513:28)
    at node:internal/child_process:878:39
    at processTicksAndRejections (node:internal/process/task_queues:78:11) {
  errno: -32,
  code: 'EPIPE',
  syscall: 'write'
}
2023-01-25T20:00:18: PM2 log: ===============================================================================
2023-01-25T20:00:18: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2023-01-25T20:00:18: PM2 log: Time                 : Wed Jan 25 2023 20:00:18 GMT+0000 (Coordinated Universal Time)
2023-01-25T20:00:18: PM2 log: PM2 version          : 5.2.2
2023-01-25T20:00:18: PM2 log: Node.js version      : 16.18.1
2023-01-25T20:00:18: PM2 log: Current arch         : x64
2023-01-25T20:00:18: PM2 log: PM2 home             : /root/.pm2
2023-01-25T20:00:18: PM2 log: PM2 PID file         : /root/.pm2/pm2.pid
2023-01-25T20:00:18: PM2 log: RPC socket file      : /root/.pm2/rpc.sock
2023-01-25T20:00:18: PM2 log: BUS socket file      : /root/.pm2/pub.sock
2023-01-25T20:00:18: PM2 log: Application log path : /root/.pm2/logs
2023-01-25T20:00:18: PM2 log: Worker Interval      : 30000
2023-01-25T20:00:18: PM2 log: Process dump file    : /root/.pm2/dump.pm2
2023-01-25T20:00:18: PM2 log: Concurrent actions   : 2
2023-01-25T20:00:18: PM2 log: SIGTERM timeout      : 1600
2023-01-25T20:00:18: PM2 log: ===============================================================================
2023-01-25T20:00:18: PM2 log: App [pm2-logrotate:0] starting in -fork mode-
2023-01-25T20:00:18: PM2 log: App [pm2-logrotate:0] online
2023-01-25T20:00:18: PM2 log: Stopping app:pm2-logrotate id:0
2023-01-25T20:00:18: PM2 log: App [pm2-logrotate:0] exited with code [0] via signal [SIGINT]
2023-01-25T20:00:19: PM2 log: pid=1679090 msg=process killed
2023-01-25T20:00:19: PM2 log: PM2 successfully stopped

How could we reproduce this issue?

I am not quite sure how to produce/avoid this issue.

Supporting information

$ pm2 report

--- PM2 report ----------------------------------------------------------------
Date                 : Thu Jan 26 2023 11:03:27 GMT+0000 (Coordinated Universal Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 5.2.2
node version         : 16.18.1
node path            : not found
argv                 : /usr/bin/node,/usr/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
uptime               : 208min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 5.2.2
node version         : 16.18.1
node path            : not found
argv                 : /usr/bin/node,/usr/bin/pm2,report
argv0                : node
user                 : root
uid                  : 0
gid                  : 0
===============================================================================
--- System info --------------------------------------------
arch                 : x64
platform             : linux
type                 : Linux
cpus                 : Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz
cpus nb              : 32
freemem              : 63062401024
totalmem             : 67530924032
home                 : /root
===============================================================================
--- PM2 list -----------------------------------------------
**Redacted**
Module
┌────┬──────────────────────────────┬───────────────┬──────────┬──────────┬──────┬──────────┬──────────┬──────────┐
│ id │ module                       │ version       │ pid      │ status   │ ↺    │ cpu      │ mem      │ user     │
├────┼──────────────────────────────┼───────────────┼──────────┼──────────┼──────┼──────────┼──────────┼──────────┤
│ 0  │ pm2-logrotate                │ 2.7.0         │ 1689056  │ online   │ 0    │ 0.6%     │ 57.4mb   │ root     │
└────┴──────────────────────────────┴───────────────┴──────────┴──────────┴──────┴──────────┴──────────┴──────────┘
===============================================================================
--- Daemon logs --------------------------------------------
/root/.pm2/pm2.log last 20 lines:
PM2        | 2023-01-26T11:00:14: PM2 log: App name:app.compiled id:33 disconnected
PM2        | 2023-01-26T11:00:14: PM2 log: App [app.compiled:33] exited with code [0] via signal [SIGINT]
PM2        | 2023-01-26T11:00:14: PM2 log: App [app.compiled:33] starting in -cluster mode-
PM2        | 2023-01-26T11:00:15: PM2 log: App [app.compiled:33] online
PM2        | 2023-01-26T11:00:17: PM2 log: App name:app.compiled id:35 disconnected
PM2        | 2023-01-26T11:00:17: PM2 log: App [app.compiled:35] exited with code [0] via signal [SIGINT]
PM2        | 2023-01-26T11:00:17: PM2 log: App [app.compiled:35] starting in -cluster mode-
PM2        | 2023-01-26T11:00:17: PM2 log: App [app.compiled:35] online
PM2        | 2023-01-26T11:00:19: PM2 log: App name:app.compiled id:36 disconnected
PM2        | 2023-01-26T11:00:19: PM2 log: App [app.compiled:36] exited with code [0] via signal [SIGINT]
PM2        | 2023-01-26T11:00:19: PM2 log: App [app.compiled:36] starting in -cluster mode-
PM2        | 2023-01-26T11:00:19: PM2 log: App [app.compiled:36] online
PM2        | 2023-01-26T11:00:22: PM2 log: App name:app.compiled id:3 disconnected
PM2        | 2023-01-26T11:00:22: PM2 log: App [app.compiled:3] exited with code [0] via signal [SIGINT]
PM2        | 2023-01-26T11:00:22: PM2 log: App [app.compiled:3] starting in -cluster mode-
PM2        | 2023-01-26T11:00:22: PM2 log: App [app.compiled:3] online
PM2        | 2023-01-26T11:01:03: PM2 log: App name:app.compiled id:32 disconnected
PM2        | 2023-01-26T11:01:03: PM2 log: App [app.compiled:32] exited with code [0] via signal [SIGINT]
PM2        | 2023-01-26T11:01:03: PM2 log: App [app.compiled:32] starting in -cluster mode-
PM2        | 2023-01-26T11:01:03: PM2 log: App [app.compiled:32] online

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Node.js cluster and IPC stack traces, then inspect the PM2 entry points named in God.js and @pm2/io, including IPCTransport.js, configuration.js, notify.js, featureManager.js, and pmx.js. Compare the EPIPE sequence with the preceding Cannot find module '/undefined' failure; done means the cause is identified and PM2 no longer stops unexpectedly in this sequence.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, nodejs
Domain
backend, cli
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.