Automattic / Automattic/kue

Delayed job processed more than once

Open
#1,133 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

It's a bit hard to reproduce, and I'm not 100% sure what to share exactly.

Setup: We have one large redis in the middle, and a variety of different queues and workers connected. Each queue is being fed by at least 10 workers, and processed by another 10.

Now, I do have one specific type of job, that's enqueued with a delay (e.g. 70 hours). This job is also a bit longer, takes on average 1.8 seconds. And this is regularly being processed twice in parallel, by different workers, maybe even on different machines.

For example, I log the start of the job and got:
```
2017-11-16 10:56:12.733 - Job 2494621079...
2017-11-16 10:56:12.180 - Job 2494621079...
```

Is there anything I can do about this, or are we just cratching the top-line and may need to move to another queue system?

Contributor guide

No contributing guide indexed for this repository

Research direction

No source files, tests, or entry points are named. Start by reproducing the delayed 70-hour job with the described shared Redis, multiple queues, and parallel workers, then trace delayed-job promotion and worker processing; done means identifying the duplicate-processing cause and verifying that the same job is not run concurrently.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.