spring-cloud / spring-cloud/spring-cloud-stream
Allow to distribute consumer group messages evenly when running multiple servers with multiple consumers each (set consumer priority to thread idx ?)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 1.1k
- Forks
- 646
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 8
Description
Describe the issue
When using multiple consumers on multiple servers, messages in a consumer groups are not evenly distributed across servers.
To Reproduce
Steps to reproduce the behavior:
using a consumer group where message handling is a computationaly intensive operation (maybe 10s of cpu), start 2 servers with 2 consumers each (e.g. SMLC concurrency=2). Publish 2 messages. Notice that both message are handled by the same server going 200% cpu, whereas the other server is idle. Also, even for small loads, servers must have enough ram to handle all the messages in parallel.
Version of the framework
2023.0.1
Expected behavior
Allow to setup evenly distributed messages across servers each running multiple consumers
For example, if each thread could set it's consumerpriority to it's index, rabbitmq would only distribute messages to low threads when higher threads are actually running
Or maybe another idea, like dynamically creating consumers under load ? But it's harder to control when its scaling down.
or create manually N bindings each with a concurrency of 1 and running the same consumer bean ?
Additional context
related issue in spring-amqp https://github.com/spring-projects/spring-amqp/issues/3092
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the SMLC concurrency=2 behavior described in the reproduction and read the related spring-amqp issue 3092. Compare the proposed consumer-priority, dynamic-consumer, and multiple-binding approaches, then define a design that can distribute consumer-group messages across servers and specify how scaling down is controlled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, rabbitmq, spring
- Domain
- distributed-systems, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100