Automattic / Automattic/kue

Same Job Repeating Hundreds of Times

Open
#899 12 comments 3 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

I am seeing sporadic issues where jobs repeat hundreds of times. It will be the same job ID and I have confirmed that I'm only adding it to the queue once. What happens is that jobs will process correctly for some time and then, randomly it seems, a single job will process 400+ times and then things return to normal.

The app is on AWS and there are two instances on which Kue runs. I noticed in the docs (https://github.com/Automattic/kue#parallel-processing-with-cluster) it shows how to ensure only one instance of the web app runs by checking `cluster.isMaster` and then only starting the Kue web app on that process. While we aren't using the cluster module, we do have two separate boxes (AWS instances) and right now they both run the web app. E.g. they both run `kue.app.listen(config.port);`. Is it problematic to run that multiple times? Could that be the cause of this issue?

If not, is there anything else that might cause this?

Thanks!

Contributor guide

No contributing guide indexed for this repository

Research direction

No source file or test is identified. Start by reproducing the repeated-job behavior with Kue running on both AWS instances, then compare that setup with the documented cluster.isMaster pattern and the kue.app.listen(config.port) entry point. Done means identifying a reproducible cause and documenting a concrete fix or configuration change.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, javascript, node.js, redis
Domain
backend, cloud, distributed-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
15/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.