apache / apache/rocketmq

[Bug] TimelineRollService may repeatedly roll the same timer messages

Open
#11,156 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
22.6k
Forks
12k
Avg merge
3d 1h
Merged PRs (30d)
27

Description

### Before Creating the Bug Report

- [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions).

- [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate.

- [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

### Runtime platform environment

all environment

### RocketMQ version

develop

### JDK Version

1.8

### Describe the Bug

`TimelineRollService` used a fixed sleep plus a scan window of `[now + rollRange, now + rollRange + timerMaxDelaySec]`.

Two issues follow from that:

1. The window is much larger than the interval, so the same not-yet-expired timer message stays in range and is rolled back to `TIMER_TOPIC` many times.
2. The next window is computed from `System.currentTimeMillis()` after sleep. If a scan is delayed, the next round can skip or rescan the same delayTime range. The progress was also only in memory, so a restart could roll the same messages again.

### Steps to Reproduce

Enable the rocksdb scheduled messages, send a scheduled message, and you can change the roll interval to execute every minute. This way, you can verify more quickly.

### What Did You Expect to See?

Set a reasonable roll interval and scan range, and simultaneously record checkpoints to ensure that messages are not lost and no invalid duplicate deliveries occur.

### What Did You See Instead?

Drive roll by a persisted RocksDB checkpoint (`timeline_roll_checkpoint`) instead of a fixed sleep:

- Each round scans `[checkpoint, checkpoint + interval)`.
- After a successful scan, advance and persist the checkpoint.
- The next due time is `checkpoint + interval - timerMaxDelaySec`. Trigger 1s early; if it is not due yet, poll with at most 1s wait.
- If the service is behind, scan the next window immediately so delayed work does not leave a gap.

## Test plan

- [ ] `MessageRocksDBStorageTest#testWriteAndGetRollCheckpoint`
- [ ] `MessageRocksDBStorageTest#testScanAdjacentWindowsNoOverlap`
- [ ] Send a long-delay timer message (e.g. 3h, `timerMaxDelaySec` a bit larger) and confirm it is rolled at most once
- [ ] Slow down or restart broker during roll and confirm the next scan continues from the checkpoint without skipping or duplicating the previous window

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading TimelineRollService and the checkpoint-related paths covered by MessageRocksDBStorageTest. Run testWriteAndGetRollCheckpoint and testScanAdjacentWindowsNoOverlap first, then verify the long-delay and restart scenarios described in the test plan. Done means adjacent checkpoint windows do not overlap or leave gaps, and delayed timer messages are not rolled more than once.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.