Automattic / Automattic/kue

Kue job concurrency not being honored

Open
#1,011 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

Hi there. I'm running into a weird issue where my worker processes are not honoring concurrency limits. My understanding is that the default behavior of Kue - when a concurrency value is not supplied - is to run a given job from start to success before handling the next job of that type in the queue.

```
// worker.js
...
const queue = require('./lib/queue') // Reference to my library file for Kue functionality
...
```

```
// ./lib/queue.js
...
function _createJobWithNameAndOpts (name, opts, cb) {
if (!name) { return cb() }
if (!opts) { opts = {} }

console.log('Creating new job \'' + name + '\' with opts ' + JSON.stringify(opts))

var job = jobs.create(name, opts)
var execute = require('./' + name).process

job
.on('enqueue', function () {
var message = 'QUEUED: Job ' + job.id + ' added to queue (' + dateFormatter.getFormattedTimestampForConsoleOutput() + ')'
if (VERBOSE) { console.log(message) }
})
.on('start', function () {
var message = 'START: Job ' + job.id + ' has started (' + dateFormatter.getFormattedTimestampForConsoleOutput() + ')'
if (VERBOSE) { console.log(message) }
})
.on('complete', function () {
var message = 'SUCCESS: Job ' + job.id + ' completed (' + dateFormatter.getFormattedTimestampForConsoleOutput() + ')'
if (VERBOSE) { console.log(message) }
})
.on('failed', function () {
var message = 'FAILURE: Job ' + job.id + ' did not complete (' + dateFormatter.getFormattedTimestampForConsoleOutput() + ')'
if (VERBOSE) { console.log(message) }
})


jobs.process(name, 1, execute) // The same as jobs.process(name, execute)

job.save()
return cb()
}

module.exports.createJobWithNameAndOpts = _createJobWithNameAndOpts
```

In one example file - where I pass in `name` as `invoice` - the ./invoice.js exported process method is one that simply simulates a process that takes five seconds to complete:
```
...
module.exports.process = function (job, done) {
var message = 'PROCESSING: ' + TASK_NAME + ' ' + job.id + ' is ready to be processed (' + dateFormatter.getFormattedTimestampForConsoleOutput() + ')'
console.log(message)
setTimeout(function () {
return done()
}, 5000)
}
...
```

From what I understand, I should see logging output where one process starts and completes successfully before a second process kicks off. However, here is what happens when I queue up five (5) processes:
```
13:03:14 web.1 | QUEUED: Job 2 added to queue (2017-01-04 13:03:14:896)
13:03:14 web.1 | QUEUED: Job 3 added to queue (2017-01-04 13:03:14:896)
13:03:14 web.1 | QUEUED: Job 4 added to queue (2017-01-04 13:03:14:896)
13:03:14 web.1 | QUEUED: Job 5 added to queue (2017-01-04 13:03:14:896)
13:03:14 web.1 | QUEUED: Job 6 added to queue (2017-01-04 13:03:14:896)
13:03:14 web.1 | PROCESSING: invoice 2 is ready to be processed (2017-01-04 13:03:14:958)
13:03:14 web.1 | PROCESSING: invoice 3 is ready to be processed (2017-01-04 13:03:14:958)
13:03:14 web.1 | PROCESSING: invoice 4 is ready to be processed (2017-01-04 13:03:14:959)
13:03:14 web.1 | PROCESSING: invoice 5 is ready to be processed (2017-01-04 13:03:14:959)
13:03:14 web.1 | PROCESSING: invoice 6 is ready to be processed (2017-01-04 13:03:14:959)
13:03:15 web.1 | START: Job 2 has started (2017-01-04 13:03:15:018)
13:03:15 web.1 | START: Job 3 has started (2017-01-04 13:03:15:018)
13:03:15 web.1 | START: Job 4 has started (2017-01-04 13:03:15:019)
13:03:15 web.1 | START: Job 5 has started (2017-01-04 13:03:15:019)
13:03:15 web.1 | START: Job 6 has started (2017-01-04 13:03:15:019)
13:03:20 web.1 | SUCCESS: Job 2 completed (2017-01-04 13:03:20:142)
13:03:20 web.1 | SUCCESS: Job 3 completed (2017-01-04 13:03:20:142)
13:03:20 web.1 | SUCCESS: Job 4 completed (2017-01-04 13:03:20:142)
13:03:20 web.1 | SUCCESS: Job 5 completed (2017-01-04 13:03:20:142)
13:03:20 web.1 | SUCCESS: Job 6 completed (2017-01-04 13:03:20:142)
```

What could be the issue here? Shouldn't I see a `SUCCESS: Job 2 ...` message before job 3 is processed/started? Each invoice job will take five seconds to run, and if it's only supposed to run them one at a time, why is it starting them at once? I was able to confirm this by running 50 jobs against this, and sure enough all 50 start at the same time. I wouldn't want to have 50 connections attempting to be opened to the database cluster - the goal is to have them fire sequentially =)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the jobs.process(name, 1, execute) registration shown in lib/queue.js and compare it with the worker.js setup. Reproduce the behavior with the five-second process function in invoice.js, then verify why five jobs begin together instead of one completing before the next starts.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.