Process only one job of a given type at a time across containers
- Dominant language
- JavaScript
- Stars
- 9.4k
- Forks
- 858
- PR merge metrics
- No merged PRs in 30d
Description
Is there a way to process only one job of a given type at a time across containers?
We have a number of job types that we want to process across a handful of containers. It is important that we process the jobs of each type in the order they were added to the queue, so we want only one job from a given job type to be processed at any given time across all the containers. However, we also want to be able to scale horizontally, so while container A is processing one job type, container B can process jobs of another type.
We can use the second parameter on `queue.process` to control concurrency within a container. Is there a way we can control concurrency across containers?
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue names queue.process and asks about cross-container concurrency; start by reading that API and the queue's Redis-backed worker behavior. Done would require a documented or implemented way to serialize jobs by type across containers while allowing different types to run in parallel.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- javascript, node.js, redis
- Domain
- backend, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100