Unitech / Unitech/pm2

PM2 restarts app when a subprocess was killed by the OOM killer

Open
#5,941 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
JavaScript
Stars
43.3k
Forks
2.7k
PR merge metrics
No merged PRs in 30d

Description

I have a node app running via pm2. The app itself spawns some processes like imagemagick's convert to convert some images in a folder. It can happen, that the convert call may use too much memory and get killed by the systems OOM killer with the message:

Out of memory: Killed process 111337 (convert) total-vm:562060kB, anon-rss:416096kB, file-rss:4kB, shmem-rss:67340kB, UID:0 pgtables:1060kB oom_score_adj:1000

This is fine. In this case my app would log an error that this process was not successful.

However, right after the OOM killer killed the sub process, pm2 kills my node app with the following message:

systemd[1]: pm2-root.service: A process of this unit has been killed by the OOM killer.
pm2[111342]: [PM2] Applying action deleteProcessId on app [all](ids: [ 0, 1, 2 ])

It seems, that pm2 detects that the OOM killer killed a sub process and then calls an internal method deleteProcessId on all apps.
Because of this, my app cannot log any error. It does get restarted, but then tries the same file until it gets killed again.
Is this the default behavior of pm2? Can this behavior be bypassed?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the reported OOM-killer sequence with a PM2-managed Node.js app and a killed subprocess. Inspect the handling around the mentioned deleteProcessId action and compare it with the systemd OOM message. Done means establishing whether this is expected behavior and identifying a supported way to bypass it or documenting that none exists.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.