apache / apache/eventmesh

Suggesting one consumer each runtime and self-management offset mode

Open
#471 3 comments 1 reaction 0 assignees View on GitHub
discussion enhancement
Dominant language
Java
Stars
1.8k
Forks
658
Avg merge
11h 29m
Merged PRs (30d)
52

Description

## Enhancement Request

### Is your enhancement request related to a problem? Please describe

EventMesh currently chooses a consumer group-level agent, that is, there is only one real Consumer on a single EventMesh instance of a consumer group.
The number of clients that a single EventMesh instance can support is limited by the number of consumers and the average number of clients in every consumer group.
Each consumer has a private resource pool. In general, this resource pool includes but is not limited to a collection of threads, such as threads used to obtain messages and consumption, and some timing threads used to access the registry. When there are a limited number of consumers After being created on the EventMesh instance, a resource explosion will occur.

The producer has simple logic, takes up less resources and can be ignored.

like this, when several consumers are created, the JVM will not have enough threads, and the memory resources will run stably
![image](https://user-images.githubusercontent.com/1085564/126992485-e258435e-1321-4072-846a-b8f5e9bdcdb3.png)

Scene P1 and Scene P2 show the number of clients that a single EventMesh instance can support in two cases (on average, a single consumer group has more clients and fewer clients).

![EventMesh Runtime](https://user-images.githubusercontent.com/1085564/126992524-3e4e95cc-eec5-4ac5-bb99-6d5a7dd4fe07.png)

Obviously, in terms of quantity, scenario 1 is better In scene 2.

### Describe the solution you'd like
Global consumer, a single EventMesh instance has one and only one singleton Consumer & Producer, this singleton Consumer&Producer is responsible for all clients connected to the EventMesh to send and receive events, if a single EventMesh instance only has one global consumer,

include two parts
1. manage meta-data about client-group mapping
~ meta-data table like example
~ sync meta-data with broadcast
![image](https://user-images.githubusercontent.com/1085564/126994788-074d26ff-fe9f-47c6-a0e8-730bcafc2114.png)

2. local ack manage
~ The consumption progress of EventMesh is managed separately from the consumption progress of the actual consumer group

![image](https://user-images.githubusercontent.com/1085564/126996813-2e4af5fb-a2ed-4565-be32-9ae16bf4eb79.png)

### Describe alternatives you've considered

Contributor guide

Open the contributing guide

Research direction

Start by reading the EventMesh consumer and producer runtime behavior described in the issue, especially consumer-group ownership, client-group metadata, broadcast synchronization, and local acknowledgement management. Define the singleton consumer/producer model and its metadata and acknowledgement requirements first; done means one runtime can serve all connected clients without per-consumer resource explosion while preserving consumption progress.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.