apache / apache/rocketmq

[Bug] Timer dequeue may get stuck after CommitLog truncation due to stale TimerLog references

Open
#10,668 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
22.6k
Forks
12k
Avg merge
3d 1h
Merged PRs (30d)
27

Description

### Before Creating the Bug Report

- [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions).

- [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate.

- [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

### Runtime platform environment

ubuntu 24.04

### RocketMQ version

branch: develop version: 5.5.0 Git commit id: a6fb9e2fa0d4e446c88b0051cd4fac233dcee9ec

### JDK Version

JDK 8u202

### Describe the Bug

After a slave truncates its CommitLog to a smaller offset, Timer-related data structures such as TimerLog and TimerWheel may still retain records that reference CommitLog offsets beyond the truncation point.

These stale references can cause the Timer dequeue pipeline to read an unrelated message and retry it indefinitely.

Assume TimerLog contains a record like:

```text
TimerLog entry -> CommitLog offset P
```

Originally, offset `P` contains a valid `TIMER_TOPIC` message.

After CommitLog is truncated to offset `T`, where `T < P`, the TimerLog entry is not removed. When CommitLog data is transferred again, offset `P` may be overwritten by a completely different message.

The resulting state is:

```text
TimerLog entry -> CommitLog offset P
CommitLog[P] -> non-TIMER_TOPIC message
```

When Timer dequeue processes this TimerLog entry, it reads the new message at `P` and treats it as the original Timer wrapper message.

The new message may not contain required Timer properties such as:

```text
PROPERTY_REAL_TOPIC
PROPERTY_REAL_QUEUE_ID
```

This can cause an exception such as:

```text
java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java:542)
at java.lang.Integer.parseInt(Integer.java:615)
at org.apache.rocketmq.store.timer.TimerMessageStore.convertMessage(TimerMessageStore.java)
at org.apache.rocketmq.store.timer.TimerMessageStore.convert(TimerMessageStore.java)
at org.apache.rocketmq.store.timer.TimerMessageStore$TimerDequeuePutMessageService.run(TimerMessageStore.java)
```

With `timerSkipUnknownError=false`, `TimerDequeuePutMessageService` keeps retrying the same `TimerRequest`.

The TimerLog record is not repeatedly read. Instead, the already-created `TimerRequest` remains in the internal retry loop and fails continuously.

## Impact

A single stale TimerLog reference can prevent the current Timer slot from completing.

As a result:

- Timer dequeue progress cannot advance.
- Subsequent scheduled messages cannot be delivered.
- Retry messages using the Timer mechanism, including POP revive retry delivery, may remain blocked.
- The broker continuously reports the same exception.

### Steps to Reproduce

This issue can be reproduced with a simulated `MessageStore` without constructing the complete HA failure sequence.

1. Create a `TimerMessageStore` with a temporary TimerLog/TimerWheel directory and a mocked `MessageStore`.

2. Keep the default configuration:

```properties
timerSkipUnknownError=false
```

3. Create a valid Timer message and append a TimerLog record that references a simulated CommitLog offset:

```text
offsetPy = P
sizePy = S
```

Ensure the corresponding TimerWheel slot references this TimerLog record.

4. Configure the mocked `MessageStore` so that reading `(P, S)` now returns a valid ordinary message instead of the original `TIMER_TOPIC` message.

The replacement message should:

- Have a non-`TIMER_TOPIC` topic.
- Not contain `PROPERTY_REAL_TOPIC`.
- Not contain `PROPERTY_REAL_QUEUE_ID`.

This simulates the state after CommitLog truncation and subsequent HA data overwrite.

5. Enable Timer dequeue and advance the Timer clock until the slot containing the simulated stale record is processed.

6. Run `TimerDequeueGetMessageService` and `TimerDequeuePutMessageService` in separate threads with a test timeout.

7. Observe that the replacement message is passed to `convertMessage()` and triggers:

```text
java.lang.NumberFormatException: null
at java.lang.Integer.parseInt(Integer.java)
at org.apache.rocketmq.store.timer.TimerMessageStore.convertMessage(TimerMessageStore.java)
at org.apache.rocketmq.store.timer.TimerMessageStore.convert(TimerMessageStore.java)
at org.apache.rocketmq.store.timer.TimerMessageStore$TimerDequeuePutMessageService.run(TimerMessageStore.java)
```

8. Verify that:

- The same `TimerRequest` is retried repeatedly.
- The request latch is not completed.
- Timer read progress does not advance.
- A valid Timer message scheduled after this record is not delivered.

9. Stop the Timer service explicitly at the end of the test to avoid leaving the retry thread running.

### What Did You Expect to See?

The message referenced by TimerLog should be validated before conversion. If it is no longer a valid `TIMER_TOPIC` message, the stale request should be released and Timer dequeue should continue.

### What Did You See Instead?

The ordinary message is treated as a Timer wrapper message. Conversion fails repeatedly, and the current Timer slot cannot complete.

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with TimerMessageStore.java, especially convertMessage(), convert(), TimerDequeueGetMessageService, and TimerDequeuePutMessageService. Trace how a TimerRequest is retried when the referenced message is no longer a TIMER_TOPIC message, then reproduce the stale-reference scenario with the mocked MessageStore and separate dequeue threads. Done means the stale request is released, timer read progress advances, and later scheduled messages are delivered without repeated exceptions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.