PM2 Status Online but Application Unresponsive Intermittently

The application is running using PM2, and the status consistently shows as "online" in the pm2 list output. However, the application becomes unresponsive at random, unexpected times. This issue persists even after enabling --watch for file changes. The problem occurs intermittently without clear error logs or indication in PM2 logs. The PM2 processes are listed as running, but the application is not responsive.

Steps to Reproduce

1. Start the application using PM2 in cluster mode with the --watch flag.
2. Allow the application to run for a period of time (seems to happen randomly).
3. Observe that despite showing "online," the application becomes unresponsive.
4. Logs do not indicate any immediate issues or failures, and restarts do not resolve the issue.

Expected Behavior

The application should remain responsive as long as it is running and detected as "online" by PM2. When file changes occur, PM2 should restart the application smoothly, and the app should continue to function without interruptions.

Actual Behavior

The application becomes unresponsive intermittently, even though:

• PM2 status shows as "online."
• The CPU and memory usage remain low, with minimal spikes.
• The app instances are running, but no longer handle requests or function correctly.

PM2 Report

--- PM2 report ----------------------------------------------------------------
Date                 : Mon Sep 30 2024 07:36:24 GMT+0000 (Coordinated Universal Time)
===============================================================================
--- Daemon -------------------------------------------------
pm2d version         : 5.4.2
node version         : 18.19.1
node path            : not found
argv                 : /usr/bin/node,/usr/local/lib/node_modules/pm2/lib/Daemon.js
argv0                : node
user                 : ubuntu
uid                  : 1000
gid                  : 1000
uptime               : 14607min
===============================================================================
--- CLI ----------------------------------------------------
local pm2            : 5.4.2
node version         : 18.19.1
node path            : /usr/local/bin/pm2
argv                 : /usr/bin/node,/usr/local/bin/pm2,report
argv0                : node
user                 : ubuntu
uid                  : 1000
gid                  : 1000
===============================================================================
--- System info --------------------------------------------
arch                 : arm64
platform             : linux
type                 : Linux
cpus                 : Neoverse-N1
cpus nb              : 2
freemem              : 3075633152
totalmem             : 4018012160
home                 : /home/ubuntu
===============================================================================
--- PM2 list -----------------------------------------------
┌────┬────────┬─────────────┬─────────┬─────────┬──────────┬────────┬──────┬───────────┬──────────┬──────────┬──────────┬──────────┐
│ id │ name   │ namespace   │ version │ mode    │ pid      │ uptime │ ↺    │ status    │ cpu      │ mem      │ user     │ watching │
├────┼────────┼─────────────┼─────────┼─────────┼──────────┼────────┼──────┼───────────┼──────────┼──────────┼──────────┼──────────┤
│ 0  │ app    │ default     │ 1.0.0   │ cluster │ 37154    │ 74m    │ 6    │ online    │ 0.3%     │ 129.9mb  │ ubuntu   │ enabled  │
│ 1  │ app    │ default     │ 1.0.0   │ cluster │ 37155    │ 74m    │ 6    │ online    │ 0.2%     │ 131.1mb  │ ubuntu   │ enabled  │
│ 2  │ app    │ default     │ 1.0.0   │ cluster │ 37176    │ 74m    │ 6    │ online    │ 0.2%     │ 130.0mb  │ ubuntu   │ enabled  │
│ 3  │ app    │ default     │ 1.0.0   │ cluster │ 37177    │ 74m    │ 6    │ online    │ 0.2%     │ 130.7mb  │ ubuntu   │ enabled  │
│ 4  │ app    │ default     │ 1.0.0   │ cluster │ 37203    │ 74m    │ 6    │ online    │ 0.2%     │ 132.3mb  │ ubuntu   │ enabled  │
└────┴────────┴─────────────┴─────────┴─────────┴──────────┴────────┴──────┴───────────┴──────────┴──────────┴──────────┴──────────┘

Environment Details

• PM2 Version: 5.4.2
• Node Version: 18.19.1
• Architecture: arm64
• Platform: Linux
• Memory: 4GB total, ~3GB free
• App Running Mode: Cluster
• Watch Mode: Enabled

Daemon Logs

The last 20 lines of PM2 logs don’t show anything significant at the time of the issue.

/home/ubuntu/.pm2/pm2.log last 20 lines:

Additional Notes

• The application has been restarted 6 times (↺ 6), but the restarts don’t seem to resolve the issue.
• The application becomes unresponsive without any significant CPU or memory spikes.
• We have already tried increasing resource limits and switching from --watch to pm2 reload at intervals, but the problem persists.

Expected Fixes

• Debugging why the application becomes unresponsive despite showing "online."
• Ensuring that --watch properly detects changes and restarts the application as needed.
• Any additional suggestions for improving cluster mode reliability.

Start with the PM2 report, the pm2 list cluster/watch output, and /home/ubuntu/.pm2/pm2.log, comparing daemon and application behavior around an unresponsive period. Reproduce the intermittent failure if possible and identify why processes remain marked online; done means the cause is explained and a verified fix or diagnostic is documented.