SIGABRT (signal 6) after a job is killed by the Linux OOM killer
Nobody has claimed this yet.
- Dominant language
- Ruby
- Stars
- 9.5k
- Forks
- 1.7k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 25
Description
We have a situation where certain memory-heavy tasks will cause our dedicated server to run out of memory and trigger the OOM killer.
The killer terminates the Resque job (which is fine) with a SIGKILL, but it also breaks Resque for any subsequent jobs (not so fine).
To replicate this issue I created a Resque job that purposely eats up all all available RAM and causes an OOM and gets reaped.
After it's reaped all subsequent tasks fail like this:

Strangely, the Workers tab for the worker in question shows this:

This doesn't make sense to me because all 24 of those tasks very much failed (and their failure shows up on the Failures tab)
Also, why is there a heartbeat? The PID should be dead, right?
I was hoping for some guidance on this issue. Right now we are thinking of setting up a cron to check dmesg for "process killed" and restarting Resque, but that seems like a dirty solution.
I am using version 2.2 of Resque.
Thanks!
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the memory-heavy Resque job on Linux and observe the worker after the OOM killer terminates it. Investigate why subsequent jobs fail while the Workers tab still shows a heartbeat, using the reported Failures and worker status as the completion criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, ruby
- Domain
- backend, operating-systems
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100