apache / apache/rocketmq

[Enhancement] Avoid per-message getBytes() for topic and CRC32 property key on the broker write path

Open
#11,002 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
22.6k
Forks
12k
Avg merge
3d 1h
Merged PRs (30d)
27

Description

### Before Creating the Enhancement Request

- [x] I have confirmed that this should be classified as an enhancement rather than a bug/feature.

### Summary

Cache the topic bytes in `MessageExtEncoder` and the CRC32 property-key bytes in `MessageDecoder` to avoid two per-message `String.getBytes()` allocations on the broker write path.

### Motivation

- `MessageExtEncoder.encode()` / `encodeWithoutProperties()` call `msgInner.getTopic().getBytes(UTF_8)` for every message. Broker traffic is highly repetitive per thread, and the encoder instance is already `ThreadLocal`, so the same topic string is re-encoded over and over.
- `MessageDecoder.createCrc32()` re-encodes the constant property key `MessageConst.PROPERTY_CRC32` on every message when CRC32 stamping is enabled.

### Solution

- `MessageExtEncoder`: keep a single-slot `cachedTopic`/`cachedTopicData` pair (no synchronization needed — the encoder is `ThreadLocal`). Hit returns the cached bytes; miss falls back to `getBytes` and refreshes the slot, so mixed-topic traffic is never worse than today apart from one string comparison.
- `MessageDecoder`: pre-encode `PROPERTY_CRC32` into a `private static final byte[]`.

### Verification

- `AppendCallbackTest` 4/4, `AppendPropCRCTest` 2/2 (covers the CRC32 constant path), `LmqDispatchTest` 5/5, `CompactionLogTest` 4/4, `MessageDecoderTest` 7/7; checkstyle clean.
- 4-node cluster A/B (256-thread sync producer, 1KB, per-arm clean store + restart): broker TPS and young GC per million messages are flat versus baseline — the saving (one short `byte[]` per message) is below GC-count resolution at this load, so this is a cleanup-level allocation reduction, not a measurable throughput win.

Contributor guide

Open the contributing guide

Research direction

Start by reading MessageExtEncoder.encode() and encodeWithoutProperties(), then inspect MessageDecoder.createCrc32() and the existing verification tests. Use AppendCallbackTest, AppendPropCRCTest, LmqDispatchTest, CompactionLogTest, and MessageDecoderTest to confirm behavior and checkstyle; done means the repeated byte conversions are cached without changing message encoding or CRC32 results.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems, performance
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.