apache / apache/rocketmq-clients
[Feature] Batch acknowledgements for Java gRPC PushConsumer
- Dominant language
- Java
- Stars
- 505
- Forks
- 313
- Avg merge
- 11h 28m
- Merged PRs (30d)
- 6
Description
### Programming Language of the Client
Java
### Is Your Feature Request Related to a Problem?
The Java gRPC PushConsumer currently sends one `AckMessage` RPC for every successfully consumed normal message. The wire protocol already supports repeated `AckMessageRequest.entries`, and RocketMQ Proxy can process multi-entry acknowledgement requests, but the client constructs and sends a one-entry request from `ProcessQueueImpl` for each message.
At high consume rates, the resulting ACK RPC volume consumes a significant amount of client and Proxy CPU without improving delivery semantics.
### Describe the Solution You'd Like
Add a client-local acknowledgement batcher for Java gRPC PushConsumer with these rules:
- Accumulate acknowledgements independently within each PushConsumer.
- Partition batches by endpoint and topic because one `AckMessageRequest` carries a single topic and is sent to one endpoint.
- Flush a batch when it reaches 1,024 messages or when its oldest entry has waited 5 seconds.
- Only batch messages whose elapsed time from receive/decode to successful consumption is at most 25 seconds. Slow-consumed messages should be acknowledged immediately.
- Keep FIFO acknowledgements immediate because the next ordered message depends on the acknowledgement.
- Keep Lite consumer acknowledgements on the existing immediate path.
- Map per-entry server results back to the corresponding message and preserve the existing individual retry behavior for failed or missing results.
- Flush pending and in-flight acknowledgements during PushConsumer shutdown.
### Describe Alternatives You've Considered
1. **Only enable batch ACK on the server.** This does not reduce RPC volume because the current Java PushConsumer still emits one entry per request.
2. **Batch across topics.** This is incompatible with the protocol because an `AckMessageRequest` has one topic resource.
3. **Delay every acknowledgement, including FIFO messages.** This can block ordered consumption and change existing behavior.
4. **Retry a failed batch as another batch.** Retrying each affected message through the existing single-message path is simpler and preserves current receipt-handle error semantics.
### Additional Context
A remote Kubernetes validation against a commercial RocketMQ deployment with batch ACK enabled used the medium 2B workload: 1,000 topics, 1,000 groups, 4 KiB messages, 2,000 producers, 2,000 consumers, and approximately 8,000 TPS.
In two controlled candidate windows, ACK requests averaged about 20.4 entries per RPC, reducing ACK RPC volume by approximately 95.1%. A high-density single-topic test also observed requests containing exactly 1,024 entries.
In an A-B-B-A comparison at the same throughput, total Java consumer CPU decreased by approximately 20.6%, while total Proxy CPU decreased by approximately 17.7%.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with ProcessQueueImpl and the Java gRPC PushConsumer acknowledgement path, then inspect AckMessageRequest.entries and the existing individual retry behavior. Trace PushConsumer shutdown handling before designing the batcher. Done means the stated partitioning, size and time limits, FIFO and Lite exceptions, result mapping, retries, and shutdown flushing are covered without changing ordered-consumption semantics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java
- Domain
- backend-api-design, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100