Prevent worker processing more than 1 job type at a time
- Dominant language
- JavaScript
- Stars
- 9.4k
- Forks
- 858
- PR merge metrics
- No merged PRs in 30d
Description
I am just investigating Kue as a job processor. However I am stuck with one concept:
Say I have a worker set to process 2 job types.
Code looks something like this:
```
queue.process("job1", 1, (job, done) => {
console.log("processing job 1")
})
queue.process("job2", 1, (job, done) => {
console.log("processing job 2")
})
```
The problem with this is that if a "job1" and a "job2" are added to the queue at the same time, the above process will start both in parrallel.
In reality I will have "N" job types, each with configurable parrallel runs. I don't want a separate process per job type, as this will be hard to scale. Each process should be able to handle any job type.
How can I make sure that only 1 batch of job types will be processed at a time by any one worker?
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the queue.process API and its worker/concurrency behavior, then inspect how Kue coordinates jobs through Redis. Confirm whether one worker can serialize multiple job types while preserving each type's configured parallelism; done means concurrent job1 and job2 submissions do not overlap within that worker.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js, redis
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100