Automattic / Automattic/kue

How expensive are queue types/workers?

Open
#1,098 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

I'm new to the lib and redis, so sorry if this is a noobish question, but how expensive is it to use multiple job types?

From what I understand, in Bull, for instance, every time you create a new Queue (afaik the equivalent of a type) it means you get at least 2 new connections to redis. Is this also valid for kue?

My use case is that I must do some expensive IO on multiple IoT devices - in parallel, but only one connection per device (so a queue with concurrency: 1 for each device). Normal case is a few hundred devices, worst case a few thousand. I was thinking something along these lines:

producer.js
```
const queue = kue.createQueue();
const devices = [...];
devices.forEach(deviceID => {
const job = queue.create(deviceID, options);
job.on('complete', doSomethingWithResult);
});
```

consumer.js
```
const queue = kue.createQueue();
// scan queue periodically and create a worker for the device if there's none
function scanQueue() {
queue.types(types => types.forEach(deviceID => {
if(weDontHaveAWorkerForThisYet(deviceID)) {
queue.process(deviceID, fn);
}
}));
}
setInterval(scanQueue, 5 * 1000);
```

Is this a feasible approach or am I working my way into a pit of pain and despair?

Contributor guide

No contributing guide indexed for this repository

Research direction

Review the producer.js and consumer.js examples, then inspect Kue's createQueue, process, and types entry points to determine Redis connection costs and per-device worker behavior. Done means documenting whether this approach is feasible at the stated scale and explaining its resource implications.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend, distributed-systems
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.