apache / apache/rocketmq

[Bug] Broker may persist remoting frames from independent connections in a message body

Open
#10,911 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
22.6k
Forks
12k
Avg merge
3d 1h
Merged PRs (30d)
27

Description

### Before Creating the Bug Report

- [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions).
- [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate.
- [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

### Runtime platform environment

- OS: Linux
- Multiple independent producers and consumers connect directly to the Broker. There is no proxy or load balancer on this path.
- `serverPooledByteBufAllocatorEnable=true`
- The incident occurred during normal concurrent `SEND_MESSAGE` and long-poll `PULL_MESSAGE` traffic.

### RocketMQ version

- Branch: 4.9.8
- Version: based on Apache RocketMQ 4.9.8
- Netty: 4.1.65.Final

This has not yet been reproduced on an unmodified official RocketMQ release. We are also auditing distribution-specific changes.

### JDK Version

- Compiler/runtime: JDK 8

### Describe the Bug

A rare Broker-side data corruption was observed. A persisted business message body starts with the expected JSON, but from byte offset 457 it contains multiple RocketMQ RemotingCommand frames, including request code 11 (`PULL_MESSAGE`), request code 38, and binary total-length/header-length prefixes.

The embedded requests contain metadata from unrelated consumers running in different processes on different hosts. The producer and those consumers only share the same Broker connection endpoint; they do not share a client process or sending path.

The CommitLog record was read directly and its BODYCRC was independently verified:

```text
total message record size: 2310
body length: 2063
stored BODYCRC: 348336256 (0x14c33080)
CRC32 recalculated from the stored body: 348336256
BODYCRC_MATCH=true
```

Therefore, the unexpected bytes were already part of the message body when the Broker calculated BODYCRC. This excludes query-tool decoding and post-write disk corruption.

During the exact incident second, Broker remoting logs also show a burst of decoder errors across independent remote connections, mainly `TooLongFrameException` and JSON header parse failures. A sanitized consumer request found in one decoder failure matches the request metadata embedded in the persisted business body. A separate producer connection also shows a `SEND_MESSAGE` header interrupted by a binary frame prefix and the start of a `PULL_MESSAGE` header.

This suggests corruption in the Broker remoting receive/decode path, but the exact root cause is not yet proven. Possible areas include pooled `ByteBuf` ownership/lifetime, interaction with the codec `DefaultEventExecutorGroup`, or a distribution-specific modification.

### Steps to Reproduce

There is no deterministic reproduction yet because the event is rare. The following steps reproduce the forensic verification of the observed record:

1. Run a Broker with the pooled server allocator enabled and multiple independent clients concurrently sending messages and issuing long-poll pull requests.
2. When a corrupted message is detected, resolve its physical CommitLog offset and read the message record directly from the mapped CommitLog file.
3. Parse `TOTALSIZE`, `BODYCRC`, `SYSFLAG`, and `BODYLEN`, then extract exactly `BODYLEN` bytes from the record.
4. Recalculate CRC32 over those body bytes and compare it with the stored `BODYCRC`.
5. Search the extracted body for RocketMQ RemotingCommand headers and binary frame prefixes.
6. Correlate the embedded request metadata with Broker decoder errors from other remote connections at the same timestamp.

For the observed 2063-byte body, foreign request markers occurred at these offsets:

```text
code 11: 457, 859, 905, 1663, 1801
code 38: 1353
```

All production IP addresses, message IDs, topics, consumer groups, and raw logs are omitted from this public report.

### What Did You Expect to See?

Each channel decoder should only observe bytes received from that channel. The body persisted for a `SEND_MESSAGE` request should match the producer's request body. A malformed frame on one connection should be rejected or close only that connection and must not affect another connection's request.

### What Did You See Instead?

Remoting request bytes associated with independent consumer connections appeared inside another producer's persisted business message body. Because the body was still structurally acceptable to the decoded `SEND_MESSAGE` request, the Broker stored it with a matching BODYCRC instead of rejecting it.

Some mixed frames caused decoder exceptions and connection closures, while this body-only corruption passed decoding and was persisted silently.

### Additional Context

The closest issues found were:

- #8086 and #6784: gRPC clients connecting to classic Remoting ports, causing protocol mismatch.
- #3292: a TLS and `transferMsgByHeap=false` issue on the client response path.

None of them reports frames from independent connections appearing in a persisted message body with a matching BODYCRC.

Relevant 4.9.8 code path:

- `NettyRemotingServer` uses `DefaultEventExecutorGroup` for the pipeline and optionally configures `PooledByteBufAllocator.DEFAULT`.
- Each channel receives a new `NettyDecoder` extending `LengthFieldBasedFrameDecoder`.
- `NettyDecoder` calls `RemotingCommand.decode(frame)` and releases the extracted frame in `finally`.
- `RemotingCommand.decode(ByteBuf)` copies the remaining body into a new `byte[]`.

Questions for maintainers:

1. Is there a known issue in this 4.9.8 pipeline that can cause receive-buffer data to be observed across independent channels?
2. Was a related fix made in a later RocketMQ or Netty version?
3. Are disabling the pooled server allocator, keeping decoding on the channel EventLoop, and enabling Netty leak detection appropriate diagnostic mitigations?

Contributor guide

Open the contributing guide

Research direction

Start with remoting/NettyRemotingServer, NettyDecoder, and RemotingCommand.decode(ByteBuf) in the 4.9.8 code path; review the DefaultEventExecutorGroup, pooled allocator, and frame-release behavior. Use the listed forensic CommitLog and decoder-log correlation steps under concurrent SEND_MESSAGE and PULL_MESSAGE traffic. Done means isolating the cross-channel contamination cause and establishing a reproducible verification or regression test.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.