resque / resque/resque

SIGABRT (signal 6) after a job is killed by the Linux OOM killer

Open
#1,783 7 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Ruby
Stars
9.5k
Forks
1.7k
Avg merge
1d 3h
Merged PRs (30d)
25

Description

We have a situation where certain memory-heavy tasks will cause our dedicated server to run out of memory and trigger the OOM killer.
The killer terminates the Resque job (which is fine) with a SIGKILL, but it also breaks Resque for any subsequent jobs (not so fine).

To replicate this issue I created a Resque job that purposely eats up all all available RAM and causes an OOM and gets reaped.
After it's reaped all subsequent tasks fail like this:

image

Strangely, the Workers tab for the worker in question shows this:
image

This doesn't make sense to me because all 24 of those tasks very much failed (and their failure shows up on the Failures tab)
Also, why is there a heartbeat? The PID should be dead, right?

I was hoping for some guidance on this issue. Right now we are thinking of setting up a cron to check dmesg for "process killed" and restarting Resque, but that seems like a dirty solution.

I am using version 2.2 of Resque.

Thanks!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the memory-heavy Resque job on Linux and observe the worker after the OOM killer terminates it. Investigate why subsequent jobs fail while the Workers tab still shows a heartbeat, using the reported Failures and worker status as the completion criteria.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux, ruby
Domain
backend, operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.