pm2 crashes with ENFILE error, all running apps down

Start by reading lib/God/ForkMode.js, lib/Utility.js, and lib/God/Methods.js around the reported fork and logging paths, then investigate the Linux EMFILE, ENFILE, and ENOENT conditions. Done means confirming the cause of the process-fork failure and defining a tested or documented resolution for the reported outage.

Please I need help here!

I'm using pm2 and run around 30 app on Linux with pm2 and everything run perfect, But there's an error occur every interval make all the running apps down and when I checked the logs I found this.

PM2 error: Trace: Error: Process could not be forked properly, check your system health
at /opt/apps/nodejs-16/lib/node_modules/pm2/lib/God/ForkMode.js:120:26
at wrapper (/opt/apps/nodejs-16/lib/node_modules/pm2/node_modules/async/internal/once.js:12:16)
at next (/opt/apps/nodejs-16/lib/node_modules/pm2/node_modules/async/waterfall.js:96:20)
at /opt/apps/nodejs-16/lib/node_modules/pm2/node_modules/async/internal/onlyOnce.js:12:16
at WriteStream. (/opt/apps/nodejs-16/lib/node_modules/pm2/lib/Utility.js:186:13)
at WriteStream.emit (node:events:513:28)
at WriteStream.emit (node:domain:552:15)
at node:internal/fs/streams:75:16
at FSReqCallback.oncomplete (node:fs:196:23)
at FSReqCallback.callbackTrampoline (node:internal/async_hooks:130:17)
at Object.God.logAndGenerateError (/opt/apps/nodejs-16/lib/node_modules/pm2/lib/God/Methods.js:34:15)
at /opt/apps/nodejs-16/lib/node_modules/pm2/lib/God/ForkMode.js:121:13
at wrapper (/opt/apps/nodejs-16/lib/node_modules/pm2/node_modules/async/internal/once.js:12:16)
at next (/opt/apps/nodejs-16/lib/node_modules/pm2/node_modules/async/waterfall.js:96:20)
at /opt/apps/nodejs-16/lib/node_modules/pm2/node_modules/async/internal/onlyOnce.js:12:16
at WriteStream. (/opt/apps/nodejs-16/lib/node_modules/pm2/lib/Utility.js:186:13)
at WriteStream.emit (node:events:513:28)
at WriteStream.emit (node:domain:552:15)
at node:internal/fs/streams:75:16
at FSReqCallback.oncomplete (node:fs:196:23)
2023-05-31T03:11:19: PM2 log: ===============================================================================
2023-05-31T03:11:19: PM2 log: --- PM2 global error caught ---------------------------------------------------
2023-05-31T03:11:19: PM2 log: Time : Wed May 31 2023 03:11:19 GMT+0300 (Arabian Standard Time)
2023-05-31T03:11:19: PM2 error: spawn /opt/apps/nodejs-16/bin/node EMFILE
2023-05-31T03:11:19: PM2 error: Error: spawn /opt/apps/nodejs-16/bin/node EMFILE
at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
at onErrorNT (node:internal/child_process:478:16)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
2023-05-31T03:11:19: PM2 log: ===============================================================================
2023-05-31T03:11:19: PM2 error: [PM2] Resurrecting PM2
node:events:491
throw er; // Unhandled 'error' event
^

Error: spawn node ENOENT
at Process.ChildProcess._handle.onexit (node:internal/child_process:283:19)
at onErrorNT (node:internal/child_process:478:16)
at processTicksAndRejections (node:internal/process/task_queues:83:21)
Emitted 'error' event on ChildProcess instance at:
at Process.ChildProcess._handle.onexit (node:internal/child_process:289:12)
at onErrorNT (node:internal/child_process:478:16)
at processTicksAndRejections (node:internal/process/task_queues:83:21) {
errno: -2,
code: 'ENOENT',
syscall: 'spawn node',
path: 'node',
spawnargs: [ undefined, 'update' ]
}