Workers error after shutdown
- Dominant language
- JavaScript
- Stars
- 9.4k
- Forks
- 858
- PR merge metrics
- No merged PRs in 30d
Description
During peak times, we routinely see workers error during shutdown. What appears to occur is this:
* We call kue.shutdown
* kue.shutdown calls worker.shutdown
* All workers call their callbacks
* kue.shutdown calls it's callback
* The app begins to lower, tearing down database connections
* Workers error, unable to connect to databases.
It's almost certainly a race condition. One issue I see is in worker.shutdown:
```
if( !this.running ) return _fn();
this.running = false;
// As soon as we're free, signal that we're done
if( !this.job ) {
return _fn();
}
```
Here, shutdown sets the worker state to `running = false`, and if the worker has not obtained a job, we call the callback. If the worker is in the middle of `self.getJob`, then kue.shutdown may proceed before the worker actually becomes idle. Instead, the worker will continue to obtain it's job, process it, and then go idle once it checks it's running state. Easy race condition.
Not exactly sure what a solution should be. One idea is a worker records a state of fetching a job and refers to that in worker.shutdown.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing kue.shutdown through worker.shutdown, especially the getJob path described in the report. Reproduce shutdown while a worker is fetching or processing a job, then determine when the shutdown callback currently fires. Done means shutdown does not finish until in-flight worker activity is idle and workers no longer error as database connections are torn down.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js, redis
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100