How expensive are queue types/workers?
- Dominant language
- JavaScript
- Stars
- 9.4k
- Forks
- 858
- PR merge metrics
- No merged PRs in 30d
Description
I'm new to the lib and redis, so sorry if this is a noobish question, but how expensive is it to use multiple job types?
From what I understand, in Bull, for instance, every time you create a new Queue (afaik the equivalent of a type) it means you get at least 2 new connections to redis. Is this also valid for kue?
My use case is that I must do some expensive IO on multiple IoT devices - in parallel, but only one connection per device (so a queue with concurrency: 1 for each device). Normal case is a few hundred devices, worst case a few thousand. I was thinking something along these lines:
producer.js
```
const queue = kue.createQueue();
const devices = [...];
devices.forEach(deviceID => {
const job = queue.create(deviceID, options);
job.on('complete', doSomethingWithResult);
});
```
consumer.js
```
const queue = kue.createQueue();
// scan queue periodically and create a worker for the device if there's none
function scanQueue() {
queue.types(types => types.forEach(deviceID => {
if(weDontHaveAWorkerForThisYet(deviceID)) {
queue.process(deviceID, fn);
}
}));
}
setInterval(scanQueue, 5 * 1000);
```
Is this a feasible approach or am I working my way into a pit of pain and despair?
Contributor guide
No contributing guide indexed for this repository
Research direction
Review the producer.js and consumer.js examples, then inspect Kue's createQueue, process, and types entry points to determine Redis connection costs and per-device worker behavior. Done means documenting whether this approach is feasible at the stated scale and explaining its resource implications.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js, redis
- Domain
- backend, distributed-systems
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100