spring-cloud / spring-cloud/spring-cloud-stream

Allow to distribute consumer group messages evenly when running multiple servers with multiple consumers each (set consumer priority to thread idx ?)

Open
#3,122 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

ideal-for-contribution
Dominant language
Java
Stars
1.1k
Forks
646
Avg merge
2d 3h
Merged PRs (30d)
8

Description

Describe the issue
When using multiple consumers on multiple servers, messages in a consumer groups are not evenly distributed across servers.

To Reproduce
Steps to reproduce the behavior:
using a consumer group where message handling is a computationaly intensive operation (maybe 10s of cpu), start 2 servers with 2 consumers each (e.g. SMLC concurrency=2). Publish 2 messages. Notice that both message are handled by the same server going 200% cpu, whereas the other server is idle. Also, even for small loads, servers must have enough ram to handle all the messages in parallel.

Version of the framework
2023.0.1

Expected behavior
Allow to setup evenly distributed messages across servers each running multiple consumers
For example, if each thread could set it's consumerpriority to it's index, rabbitmq would only distribute messages to low threads when higher threads are actually running

Or maybe another idea, like dynamically creating consumers under load ? But it's harder to control when its scaling down.

or create manually N bindings each with a concurrency of 1 and running the same consumer bean ?

Additional context
related issue in spring-amqp https://github.com/spring-projects/spring-amqp/issues/3092

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the SMLC concurrency=2 behavior described in the reproduction and read the related spring-amqp issue 3092. Compare the proposed consumer-priority, dynamic-consumer, and multiple-binding approaches, then define a design that can distribute consumer-group messages across servers and specify how scaling down is controlled.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, rabbitmq, spring
Domain
distributed-systems, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.