Kue job concurrency not being honored
- Dominant language
- JavaScript
- Stars
- 9.4k
- Forks
- 858
- PR merge metrics
- No merged PRs in 30d
Description
Hi there. I'm running into a weird issue where my worker processes are not honoring concurrency limits. My understanding is that the default behavior of Kue - when a concurrency value is not supplied - is to run a given job from start to success before handling the next job of that type in the queue.
```
// worker.js
...
const queue = require('./lib/queue') // Reference to my library file for Kue functionality
...
```
```
// ./lib/queue.js
...
function _createJobWithNameAndOpts (name, opts, cb) {
if (!name) { return cb() }
if (!opts) { opts = {} }
console.log('Creating new job \'' + name + '\' with opts ' + JSON.stringify(opts))
var job = jobs.create(name, opts)
var execute = require('./' + name).process
job
.on('enqueue', function () {
var message = 'QUEUED: Job ' + job.id + ' added to queue (' + dateFormatter.getFormattedTimestampForConsoleOutput() + ')'
if (VERBOSE) { console.log(message) }
})
.on('start', function () {
var message = 'START: Job ' + job.id + ' has started (' + dateFormatter.getFormattedTimestampForConsoleOutput() + ')'
if (VERBOSE) { console.log(message) }
})
.on('complete', function () {
var message = 'SUCCESS: Job ' + job.id + ' completed (' + dateFormatter.getFormattedTimestampForConsoleOutput() + ')'
if (VERBOSE) { console.log(message) }
})
.on('failed', function () {
var message = 'FAILURE: Job ' + job.id + ' did not complete (' + dateFormatter.getFormattedTimestampForConsoleOutput() + ')'
if (VERBOSE) { console.log(message) }
})
jobs.process(name, 1, execute) // The same as jobs.process(name, execute)
job.save()
return cb()
}
module.exports.createJobWithNameAndOpts = _createJobWithNameAndOpts
```
In one example file - where I pass in `name` as `invoice` - the ./invoice.js exported process method is one that simply simulates a process that takes five seconds to complete:
```
...
module.exports.process = function (job, done) {
var message = 'PROCESSING: ' + TASK_NAME + ' ' + job.id + ' is ready to be processed (' + dateFormatter.getFormattedTimestampForConsoleOutput() + ')'
console.log(message)
setTimeout(function () {
return done()
}, 5000)
}
...
```
From what I understand, I should see logging output where one process starts and completes successfully before a second process kicks off. However, here is what happens when I queue up five (5) processes:
```
13:03:14 web.1 | QUEUED: Job 2 added to queue (2017-01-04 13:03:14:896)
13:03:14 web.1 | QUEUED: Job 3 added to queue (2017-01-04 13:03:14:896)
13:03:14 web.1 | QUEUED: Job 4 added to queue (2017-01-04 13:03:14:896)
13:03:14 web.1 | QUEUED: Job 5 added to queue (2017-01-04 13:03:14:896)
13:03:14 web.1 | QUEUED: Job 6 added to queue (2017-01-04 13:03:14:896)
13:03:14 web.1 | PROCESSING: invoice 2 is ready to be processed (2017-01-04 13:03:14:958)
13:03:14 web.1 | PROCESSING: invoice 3 is ready to be processed (2017-01-04 13:03:14:958)
13:03:14 web.1 | PROCESSING: invoice 4 is ready to be processed (2017-01-04 13:03:14:959)
13:03:14 web.1 | PROCESSING: invoice 5 is ready to be processed (2017-01-04 13:03:14:959)
13:03:14 web.1 | PROCESSING: invoice 6 is ready to be processed (2017-01-04 13:03:14:959)
13:03:15 web.1 | START: Job 2 has started (2017-01-04 13:03:15:018)
13:03:15 web.1 | START: Job 3 has started (2017-01-04 13:03:15:018)
13:03:15 web.1 | START: Job 4 has started (2017-01-04 13:03:15:019)
13:03:15 web.1 | START: Job 5 has started (2017-01-04 13:03:15:019)
13:03:15 web.1 | START: Job 6 has started (2017-01-04 13:03:15:019)
13:03:20 web.1 | SUCCESS: Job 2 completed (2017-01-04 13:03:20:142)
13:03:20 web.1 | SUCCESS: Job 3 completed (2017-01-04 13:03:20:142)
13:03:20 web.1 | SUCCESS: Job 4 completed (2017-01-04 13:03:20:142)
13:03:20 web.1 | SUCCESS: Job 5 completed (2017-01-04 13:03:20:142)
13:03:20 web.1 | SUCCESS: Job 6 completed (2017-01-04 13:03:20:142)
```
What could be the issue here? Shouldn't I see a `SUCCESS: Job 2 ...` message before job 3 is processed/started? Each invoice job will take five seconds to run, and if it's only supposed to run them one at a time, why is it starting them at once? I was able to confirm this by running 50 jobs against this, and sure enough all 50 start at the same time. I wouldn't want to have 50 connections attempting to be opened to the database cluster - the goal is to have them fire sequentially =)
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the jobs.process(name, 1, execute) registration shown in lib/queue.js and compare it with the worker.js setup. Reproduce the behavior with the five-second process function in invoice.js, then verify why five jobs begin together instead of one completing before the next starts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js, redis
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100