apache / apache/rocketmq

[Bug] MessageRocksDBStorage accumulates timer-WAL flush tasks after reload

Open
#11,013 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
22.6k
Forks
12k
Avg merge
3d 1h
Merged PRs (30d)
27

Description

### Before Creating the Bug Report

- [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions).

- [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate.

- [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

### Runtime platform environment

Ubuntu 22.04.5 LTS, Linux x86_64.

### RocketMQ version

Branch: `develop`
Version: `5.5.1`
Git commit id: `e90303810a14`

### JDK Version

OpenJDK 8u502.

### Describe the Bug

`MessageRocksDBStorage` creates a `ScheduledExecutorService` and registers a
periodic timer-WAL flush task in `postLoad()`:

```java
private final ScheduledExecutorService scheduler =
Executors.newScheduledThreadPool(1);

scheduler.scheduleAtFixedRate(this::flushTimerWal, 5, 5, TimeUnit.MINUTES);
```

However, `preShutdown()` does not cancel the scheduled task or shut down the
scheduler. `AbstractRocksDBStorage.reloadRocksdb()` uses the existing
`shutdown()` followed by `start()` lifecycle. Because the same scheduler
remains active, every successful reload registers another periodic flush task
on the same executor.

This also leaves a non-daemon scheduler and its delayed task alive after the
RocksDB resources have been closed.

### Steps to Reproduce

1. Create a `MessageRocksDBStorage` with a temporary `MessageStoreConfig`.
2. Inspect its private scheduler as a `ScheduledThreadPoolExecutor` using a
focused test or debugger. After construction, one periodic task is queued.
3. Call `storage.shutdown()`.
4. Inspect the scheduler again and observe that it is not shut down and that
its queue still contains one periodic task.
5. Call `storage.start()` to simulate the lifecycle used by
`AbstractRocksDBStorage.reloadRocksdb()`.
6. Call `storage.shutdown()` and `storage.start()` once more.
7. Inspect the scheduler after each start.

The affected code shows the following stable state transition:

| Lifecycle state | Scheduler state | Queued periodic tasks |
| --- | --- | ---: |
| After construction | active | 1 |
| After first shutdown | not shut down | 1 |
| After first start | active | 2 |
| After second shutdown | not shut down | 2 |
| After second start | active | 3 |

### What Did You Expect to See?

- `storage.shutdown()` should cancel the periodic timer-WAL flush task and
shut down its scheduler.
- Each subsequent `start()` should create one scheduler with exactly one
periodic flush task.
- A scheduled task should not access RocksDB resources after they have been
closed.

### What Did You See Instead?

- The scheduler remains active after `storage.shutdown()`.
- The delayed periodic task remains queued after the RocksDB instance and its
options have been closed.
- Each shutdown/start reload adds another periodic flush task to the same
executor.
- The scheduled runnable directly accesses `db`, `flushOptions`, and
`timerCFHandle` without a lifecycle guard. If it runs during shutdown, it can
access resources that are being closed or have already been cleared.

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading MessageRocksDBStorage.postLoad() and preShutdown(), then trace the shutdown/start lifecycle through AbstractRocksDBStorage.reloadRocksdb(). Reproduce the issue with a focused MessageRocksDBStorage test using a temporary MessageStoreConfig. Done means shutdown leaves no scheduled flush task or live scheduler, and each subsequent start has exactly one periodic task.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.