Job state field is missing
- Dominant language
- JavaScript
- Stars
- 9.4k
- Forks
- 858
- PR merge metrics
- No merged PRs in 30d
Description
I am using 0.11.0, and I run a stress test where I cause my redis server to run our of memory (I am using redislab's Redis cloud with a small instant of 30MB). The test created jobs faster then the consumers were able to create them. When I reach almost 100% memory on the reds cloud instant the request to create a job start failing. After a while I stop the test, and just the consumers to consume to jobs, however about 400 jobs remain stuck.
The dashboard reported 0 jobs in active/compete/delayed/failed state.
I also run -
queue.inactiveCount( function( err, total ) { // others are activeCount, completeCount, failedCount, delayedCount
console.log( 'We found inactive jobs = ' + total );
});
queue.completeCount( function( err, total ) { // others are activeCount, completeCount, failedCount, delayedCount
console.log( 'We found complete jobs = ' + total );
});
queue.failedCount( function( err, total ) { // others are activeCount, completeCount, failedCount, delayedCount
console.log( 'We found failed jobs = ' + total );
});
queue.delayedCount( function( err, total ) { // others are activeCount, completeCount, failedCount, delayedCount
console.log( 'We found delayed jobs = ' + total );
});
And all report zero jobs. So I use a reds client to see what jobs get stuck.
I pick on of the job ids that got stuck and bellow is the payload -
Field Value
max_attempts 1
type test
created_at 1484429452217
promote_at 1484429452217
updated_at 1484429452217
priority 0
data {"index":17497}
So it seems that the field 'state' is missing, this is why the UI or the other methods did not report these jobs.
I also run #watchStuckJobs and it did not make a difference. I guess because it is looking for the state field.
So my questions-
1) How to recover from this case and restart the jobs, or delete them ?
2) How it possible that the job was create but the state field was never set ?
Please note that I use a non default prefix for the queue :
var queue = kue.createQueue({
prefix: '_test2',
redis: {
port: XXXX,
host: YYYY
}});
I noticed that when it happen I got the following error from Kue -
{ [ReplyError: OOM command not allowed when used memory > 'maxmemory'.]
command: 'SET',
args:
[ 'activeJobsTTL:lock',
'UIDCUdtJEea5OcPMSEs8pA==',
'PX',
2000,
'NX' ],
code: 'OOM' }
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the stress test against a Redis instance limited to 30 MB, then inspect the stuck job hash with a Redis client and compare it with the dashboard and count methods. Start with the reported OOM failure for the activeJobsTTL:lock SET command and the watchStuckJobs behavior. Done means the cause of the missing state is established and a tested recovery or deletion path is defined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js, redis
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100