Automattic / Automattic/kue

Inconsistencies in job's state

Open
#1,204 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

kue version: 0.11.6

I'm experiencing a weird phenomena with some of our jobs. I have jobs in `{q}:jobs:active` ZSET that have their `state` set to `failed`.
I've tried to figure out how this is possible but I couldn't. My first suspect was that there was some external restart of the process during the `job.state()` function but the `MUTLI` is used there so it shouldn't cause any inconsistencies.

There is this [queue.checkActiveJobTtl()](https://github.com/Automattic/kue/blob/master/lib/kue.js#L207) mechanism that runs every second and in our case on some events we have a lot of these inconsistent jobs and these get processed every second which is causing an unnecessary load on our servers.

The simplest solution would be to add:

```js
job._state = 'active';
```

here: https://github.com/Automattic/kue/blob/master/lib/kue.js#L245 however on one server I've noticed that we have inconsistency with jobs in the "incative" box (these are in inactive ZSET but their state is set to "failed")

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with lib/kue.js, especially queue.checkActiveJobTtl() around the linked lines, and trace how job.state() uses MULTI while jobs move between the active and inactive ZSETs. Reproduce or inspect cases where a ZSET membership disagrees with the job's failed state. Done means the inconsistent states are explained and the unnecessary processing is prevented without breaking valid job transitions.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.