Automattic / Automattic/kue

Prevent worker processing more than 1 job type at a time

Open
#1,037 8 comments 1 reaction 0 assignees View on GitHub
Discussion
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

I am just investigating Kue as a job processor. However I am stuck with one concept:

Say I have a worker set to process 2 job types.

Code looks something like this:

```
queue.process("job1", 1, (job, done) => {
console.log("processing job 1")
})
queue.process("job2", 1, (job, done) => {
console.log("processing job 2")
})
```

The problem with this is that if a "job1" and a "job2" are added to the queue at the same time, the above process will start both in parrallel.

In reality I will have "N" job types, each with configurable parrallel runs. I don't want a separate process per job type, as this will be hard to scale. Each process should be able to handle any job type.

How can I make sure that only 1 batch of job types will be processed at a time by any one worker?

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the queue.process API and its worker/concurrency behavior, then inspect how Kue coordinates jobs through Redis. Confirm whether one worker can serialize multiple job types while preserving each type's configured parallelism; done means concurrent job1 and job2 submissions do not overlap within that worker.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.