apache / apache/rocketmq

[Bug] Paginated metadata sync silently drops the last topic/subscription group when count is one over a page boundary

Open
#11,041 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
22.6k
Forks
12k
Avg merge
3d 1h
Merged PRs (30d)
27

Description

### Before Creating the Bug Report

- [X] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions).
- [X] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate.
- [X] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

### Runtime platform environment
Linux, JDK 21, develop (ff8f6f74c)

### RocketMQ version
5.x develop

### Describe the Bug

The split-metadata pagination loops terminate one entry too early. The client-side loops in `MQClientAPIImpl` break when:

```java
if (topicSeq >= totalTopicNum - 1) { // line 3125
if (groupSeq >= totalGroupNum - 1) { // line 3033
```

The broker pages `[seq, seq + maxNum)` with no overlap (TopicConfigManager#subTopicConfig / SubscriptionGroupManager#subGroupTable, `totalTopicNum = size()`), so after page k the client has fetched `k·pageSize` entries. The loop therefore stops when `k·pageSize >= N − 1`, which is already true when exactly **one** entry remains unfetched, i.e. for every N ≡ 1 (mod pageSize).

Example: 2001 topics with the default page size 2000 — page 1 returns 2000 entries, `2000 >= 2001 - 1` is true, the loop breaks, and topic #2001 is never requested. No error, no retry — silently truncated metadata.

The same off-by-one exists broker-side in `BrokerOuterAPI` (getAllTopicConfig at line 834, getAllSubscriptionGroup at line 984); `BrokerOuterAPI#getAllTopicConfig` is used by `SlaveSynchronize#syncTopicConfig`, so a slave whose master has N ≡ 1 (mod 2000) topics permanently misses the last topic (the dataVersion comparison then matches, so no resync happens).

### Steps to Reproduce

1. Set `maxPageSizeInGetMetadata` to 100 (or have 2001 topics with the default 2000).
2. Call `DefaultMQAdminExt#getAllTopicConfig` on a broker with 101 topics.
3. Result contains 100 topics — the last one is missing.

### Expected Behavior

The loop must terminate only when `seq >= totalNum`; the last entry must be fetched.

### Corresponding PR

- Fix PR: #11042 (linked with `Closes #11041` in the PR description; contains the regression test that fails before the fix and passes after it).

Contributor guide

Open the contributing guide

Research direction

Read the pagination loops in MQClientAPIImpl and BrokerOuterAPI, then trace BrokerOuterAPI#getAllTopicConfig into SlaveSynchronize#syncTopicConfig. Reproduce with maxPageSizeInGetMetadata set to 100 and 101 topics, and use the regression test in PR #11042 to verify that the final topic or subscription group is retained.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.