Automattic / Automattic/kue

Job state field is missing

Open
#1,016 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

I am using 0.11.0, and I run a stress test where I cause my redis server to run our of memory (I am using redislab's Redis cloud with a small instant of 30MB). The test created jobs faster then the consumers were able to create them. When I reach almost 100% memory on the reds cloud instant the request to create a job start failing. After a while I stop the test, and just the consumers to consume to jobs, however about 400 jobs remain stuck.

The dashboard reported 0 jobs in active/compete/delayed/failed state.

I also run -
queue.inactiveCount( function( err, total ) { // others are activeCount, completeCount, failedCount, delayedCount
console.log( 'We found inactive jobs = ' + total );
});

queue.completeCount( function( err, total ) { // others are activeCount, completeCount, failedCount, delayedCount
console.log( 'We found complete jobs = ' + total );
});

queue.failedCount( function( err, total ) { // others are activeCount, completeCount, failedCount, delayedCount
console.log( 'We found failed jobs = ' + total );
});

queue.delayedCount( function( err, total ) { // others are activeCount, completeCount, failedCount, delayedCount
console.log( 'We found delayed jobs = ' + total );
});

And all report zero jobs. So I use a reds client to see what jobs get stuck.
I pick on of the job ids that got stuck and bellow is the payload -

Field Value
max_attempts 1
type test
created_at 1484429452217
promote_at 1484429452217
updated_at 1484429452217
priority 0
data {"index":17497}

So it seems that the field 'state' is missing, this is why the UI or the other methods did not report these jobs.

I also run #watchStuckJobs and it did not make a difference. I guess because it is looking for the state field.

So my questions-
1) How to recover from this case and restart the jobs, or delete them ?
2) How it possible that the job was create but the state field was never set ?

Please note that I use a non default prefix for the queue :
var queue = kue.createQueue({
prefix: '_test2',
redis: {
port: XXXX,
host: YYYY
}});

I noticed that when it happen I got the following error from Kue -

{ [ReplyError: OOM command not allowed when used memory > 'maxmemory'.]
command: 'SET',
args:
[ 'activeJobsTTL:lock',
'UIDCUdtJEea5OcPMSEs8pA==',
'PX',
2000,
'NX' ],
code: 'OOM' }

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the stress test against a Redis instance limited to 30 MB, then inspect the stuck job hash with a Redis client and compare it with the dashboard and count methods. Start with the reported OOM failure for the activeJobsTTL:lock SET command and the watchStuckJobs behavior. Done means the cause of the missing state is established and a tested recovery or deletion path is defined.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.