Inconsistencies in job's state
- Dominant language
- JavaScript
- Stars
- 9.4k
- Forks
- 858
- PR merge metrics
- No merged PRs in 30d
Description
kue version: 0.11.6
I'm experiencing a weird phenomena with some of our jobs. I have jobs in `{q}:jobs:active` ZSET that have their `state` set to `failed`.
I've tried to figure out how this is possible but I couldn't. My first suspect was that there was some external restart of the process during the `job.state()` function but the `MUTLI` is used there so it shouldn't cause any inconsistencies.
There is this [queue.checkActiveJobTtl()](https://github.com/Automattic/kue/blob/master/lib/kue.js#L207) mechanism that runs every second and in our case on some events we have a lot of these inconsistent jobs and these get processed every second which is causing an unnecessary load on our servers.
The simplest solution would be to add:
```js
job._state = 'active';
```
here: https://github.com/Automattic/kue/blob/master/lib/kue.js#L245 however on one server I've noticed that we have inconsistency with jobs in the "incative" box (these are in inactive ZSET but their state is set to "failed")
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with lib/kue.js, especially queue.checkActiveJobTtl() around the linked lines, and trace how job.state() uses MULTI while jobs move between the active and inactive ZSETs. Reproduce or inspect cases where a ZSET membership disagrees with the job's failed state. Done means the inconsistent states are explained and the unnecessary processing is prevented without breaking valid job transitions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js, redis
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100