Automattic / Automattic/kue

Workers error after shutdown

Open
#1,144 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

During peak times, we routinely see workers error during shutdown. What appears to occur is this:
* We call kue.shutdown
* kue.shutdown calls worker.shutdown
* All workers call their callbacks
* kue.shutdown calls it's callback
* The app begins to lower, tearing down database connections
* Workers error, unable to connect to databases.

It's almost certainly a race condition. One issue I see is in worker.shutdown:
```
if( !this.running ) return _fn();
this.running = false;

// As soon as we're free, signal that we're done
if( !this.job ) {
return _fn();
}
```
Here, shutdown sets the worker state to `running = false`, and if the worker has not obtained a job, we call the callback. If the worker is in the middle of `self.getJob`, then kue.shutdown may proceed before the worker actually becomes idle. Instead, the worker will continue to obtain it's job, process it, and then go idle once it checks it's running state. Easy race condition.

Not exactly sure what a solution should be. One idea is a worker records a state of fetching a job and refers to that in worker.shutdown.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing kue.shutdown through worker.shutdown, especially the getJob path described in the report. Reproduce shutdown while a worker is fetching or processing a job, then determine when the shutdown callback currently fires. Done means shutdown does not finish until in-flight worker activity is idle and workers no longer error as database connections are torn down.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.