Multi worker-node setup failing to process when one node is removed/crashes
- Dominant language
- JavaScript
- Stars
- 9.4k
- Forks
- 858
- PR merge metrics
- No merged PRs in 30d
Description
Hi,
I have two host VMs running my Kue worker process. If all nodes are online everything works fine, but during some tests if I bring one host down (by killing/suspending the VM - ie. ungraceful worker shutdown) the system gets out of sync.
Similarly with this issue [#130](https://github.com/Automattic/kue/issues/130) any new jobs are not processed right away, it will process the first inactive job on the queue, leaving the newest ones in inactive forever, until there are new jobs created and so on. Leaving always 2 inactive jobs to be processed.
Due to the nature of my app this breaks completely the user experience. We are calling the stuck job watcher every 1s [queue.watchStuckJobs(1000)] and even then this does not fix the issue.
Is this per design? If one worker crashes unexpectedly the whole system becomes unreliable? Is there anything I can do remedy this?
Thank you!
Kind regards
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reproducing the two-host worker setup, including an ungraceful VM shutdown, and inspect queue.watchStuckJobs(1000) alongside the inactive-job behavior described here. Done means that after one worker node crashes, newly created jobs are processed without remaining inactive indefinitely.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js, redis
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100