Automattic / Automattic/kue

Change the processing concurrency

Open
#1,049 3 comments 0 reactions 0 assignees View on GitHub
Feature
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

I would like to be able to change the parallel rate increase/decrease during runtime. I have tried the following yet this is not working as expected.

Expected, it would output
running, 1
running, 1
1 second pause
running, 1
running, 1
1 second pause
running, 1
running, 2
1 second pause
running, 2
running, 2
1 second pause

Actual:
running, 1
running, 1
running, 1
running, 1
running, 1
1 second pause
running, 2
running, 2
running, 2
running, 2
running, 2
running, 2

it seems that by calling process multiple times it does not update the queue it creates multiple queues quite unexpected.

Code below:
`var kue = require('kue')
, myQ = kue.createQueue();

function resetRate( newRate ) {
myQ.process('someTask', newRate, function(job, done){
console.log("running,",job.data.data);
setTimeout(function(){
done();
},1000); //simulated delay for a task
});
};
resetRate(1); //1
resetRate(1); //1
resetRate(1); //1
resetRate(2); // now it is handling 2 items in parallel.
myQ.create('someTask',{data:1}).removeOnComplete(true).save();
myQ.create('someTask',{data:1}).removeOnComplete(true).save();
myQ.create('someTask',{data:1}).removeOnComplete(true).save();
myQ.create('someTask',{data:1}).removeOnComplete(true).save();
myQ.create('someTask',{data:1}).removeOnComplete(true).save();
myQ.create('someTask',{data:2}).removeOnComplete(true).save();
myQ.create('someTask',{data:2}).removeOnComplete(true).save();
myQ.create('someTask',{data:2}).removeOnComplete(true).save();
myQ.create('someTask',{data:2}).removeOnComplete(true).save();
myQ.create('someTask',{data:2}).removeOnComplete(true).save();
myQ.create('someTask',{data:2}).removeOnComplete(true).save();

/*
Expected, it would handle 2 events a time
Actual: it will hande 5 events at a time
*/`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by running the JavaScript reproduction in the issue and inspect how repeated myQ.process calls register workers for the same task. Trace the processing queue and concurrency handling to determine whether the runtime rate can be updated instead of creating additional workers; done means the output never exceeds the current rate and reflects later changes.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js
Domain
backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.