apache / apache/rocketmq

[Enhancement] massive memory allocation in the `parsePublishMessageQueues`

Open
#9,926 0 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Java
Stars
22.6k
Forks
12k
Avg merge
2d 20h
Merged PRs (30d)
26

Description

### Before Creating the Enhancement Request

- [x] I have confirmed that this should be classified as an enhancement rather than a bug/feature.

### Summary

During my benchmark (50K LMQ tps), the rocketmq client in my process shows massive memory allocation in the `parsePublishMessageQueues`.

Here is my profiling result about the memory allocation:

Image

### Motivation

This multiple growths of the array list introduce a great influence on the performance under the high throughput workload.

### Describe the Solution You'd Like

Here is a simple proposal:
- do pre-allocation for the returned list of the `parsePublishMessageQueues`.
- Just return the original `List` if the namespace is empty.

### Describe Alternatives You've Considered

There are more complex solutions:
- Upgrade the interface about the `MessageQueueSelector` to use `size of the message queue` and `accessor by index to MessageQueue` instead of `List`, so that no need to create another `List`.
- Use thread-local buffer array list for the parsed `List` to avoid allocating memory for any to-be-sent message.

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Research direction

Locate the Java implementation of parsePublishMessageQueues and inspect how it builds the returned List, especially when the namespace is empty. Confirm the proposed allocation changes preserve the existing behavior, then verify the result with the project’s relevant tests or a high-throughput benchmark.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, performance
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.