Automattic / Automattic/kue

Multi worker-node setup failing to process when one node is removed/crashes

Open
#982 14 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

Hi,

I have two host VMs running my Kue worker process. If all nodes are online everything works fine, but during some tests if I bring one host down (by killing/suspending the VM - ie. ungraceful worker shutdown) the system gets out of sync.

Similarly with this issue [#130](https://github.com/Automattic/kue/issues/130) any new jobs are not processed right away, it will process the first inactive job on the queue, leaving the newest ones in inactive forever, until there are new jobs created and so on. Leaving always 2 inactive jobs to be processed.

Due to the nature of my app this breaks completely the user experience. We are calling the stuck job watcher every 1s [queue.watchStuckJobs(1000)] and even then this does not fix the issue.

Is this per design? If one worker crashes unexpectedly the whole system becomes unreliable? Is there anything I can do remedy this?

Thank you!

Kind regards

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the two-host worker setup, including an ungraceful VM shutdown, and inspect queue.watchStuckJobs(1000) alongside the inactive-job behavior described here. Done means that after one worker node crashes, newly created jobs are processed without remaining inactive indefinitely.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.