Change the processing concurrency
- Dominant language
- JavaScript
- Stars
- 9.4k
- Forks
- 858
- PR merge metrics
- No merged PRs in 30d
Description
I would like to be able to change the parallel rate increase/decrease during runtime. I have tried the following yet this is not working as expected.
Expected, it would output
running, 1
running, 1
1 second pause
running, 1
running, 1
1 second pause
running, 1
running, 2
1 second pause
running, 2
running, 2
1 second pause
Actual:
running, 1
running, 1
running, 1
running, 1
running, 1
1 second pause
running, 2
running, 2
running, 2
running, 2
running, 2
running, 2
it seems that by calling process multiple times it does not update the queue it creates multiple queues quite unexpected.
Code below:
`var kue = require('kue')
, myQ = kue.createQueue();
function resetRate( newRate ) {
myQ.process('someTask', newRate, function(job, done){
console.log("running,",job.data.data);
setTimeout(function(){
done();
},1000); //simulated delay for a task
});
};
resetRate(1); //1
resetRate(1); //1
resetRate(1); //1
resetRate(2); // now it is handling 2 items in parallel.
myQ.create('someTask',{data:1}).removeOnComplete(true).save();
myQ.create('someTask',{data:1}).removeOnComplete(true).save();
myQ.create('someTask',{data:1}).removeOnComplete(true).save();
myQ.create('someTask',{data:1}).removeOnComplete(true).save();
myQ.create('someTask',{data:1}).removeOnComplete(true).save();
myQ.create('someTask',{data:2}).removeOnComplete(true).save();
myQ.create('someTask',{data:2}).removeOnComplete(true).save();
myQ.create('someTask',{data:2}).removeOnComplete(true).save();
myQ.create('someTask',{data:2}).removeOnComplete(true).save();
myQ.create('someTask',{data:2}).removeOnComplete(true).save();
myQ.create('someTask',{data:2}).removeOnComplete(true).save();
/*
Expected, it would handle 2 events a time
Actual: it will hande 5 events at a time
*/`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by running the JavaScript reproduction in the issue and inspect how repeated myQ.process calls register workers for the same task. Trace the processing queue and concurrency handling to determine whether the runtime rate can be updated instead of creating additional workers; done means the output never exceeds the current rate and reflects later changes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100