Automattic / Automattic/kue

Process only one job of a given type at a time across containers

Open
#773 7 comments 0 reactions 0 assignees View on GitHub
Question
Dominant language
JavaScript
Stars
9.4k
Forks
858
PR merge metrics
No merged PRs in 30d

Description

Is there a way to process only one job of a given type at a time across containers?

We have a number of job types that we want to process across a handful of containers. It is important that we process the jobs of each type in the order they were added to the queue, so we want only one job from a given job type to be processed at any given time across all the containers. However, we also want to be able to scale horizontally, so while container A is processing one job type, container B can process jobs of another type.

We can use the second parameter on `queue.process` to control concurrency within a container. Is there a way we can control concurrency across containers?

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue names queue.process and asks about cross-container concurrency; start by reading that API and the queue's Redis-backed worker behavior. Done would require a documented or implemented way to serialize jobs by type across containers while allowing different types to run in parallel.

Written by the indexing model from the issue text.

Assessment

Tech stack
javascript, node.js, redis
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.