apache / apache/rocketmq

[Enhancement] Optimize RocksDB ConsumeQueue iterator for sequential reads

Open
#10,432 1 comment 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Java
Stars
22.6k
Forks
12k
Avg merge
2d 20h
Merged PRs (30d)
26

Description

### Before Creating the Enhancement Request

- [x] I have confirmed that this should be classified as an enhancement rather than a bug/feature.

### Summary

Currently, when RocketMQ uses RocksDB-based ConsumeQueue, the default read path is controlled by `iteratorWhenUseRocksdbConsumeQueue`, which is enabled by default.

However, this iterator is a RocketMQ-level reusable iterator. Internally, it still calls `RocksDBConsumeQueueTable#rangeQuery`, which builds N RocksDB keys for consecutive ConsumeQueue offsets and then calls `multiGet`.

This enhancement proposes optimizing the RocksDB ConsumeQueue sequential read path by using a bounded `RocksIterator` scan for consecutive offsets, while preserving the existing external ConsumeQueue iterator semantics.

### Motivation

Using RocksDB's native iterator scan for continuous offsets to improve performance.

### Describe the Solution You'd Like

Use a bounded RocksIterator scan in RocksDBConsumeQueueTable#rangeQuery to read continuous offsets while keeping existing ConsumeQueue iterator semantics unchanged.

### Describe Alternatives You've Considered

No

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by reading RocksDBConsumeQueueTable#rangeQuery and tracing the iteratorWhenUseRocksdbConsumeQueue read path. Compare the existing multiGet behavior with a bounded RocksIterator scan for consecutive offsets, then verify that the external ConsumeQueue iterator semantics remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
database
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.