apache / apache/rocketmq

[Bug] QUERY_CORRECTION_OFFSET admin query permanently deletes the filtered groups' consumer offsets

Open
#10,987 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
22.6k
Forks
12k
Avg merge
3d 1h
Merged PRs (30d)
27

Description

### Before Creating the Bug Report

- [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions).
- [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe this is not a duplicate.
- [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

### Runtime platform environment

- OS: Linux
- Component: Broker (`ConsumerOffsetManager`)

### RocketMQ version

- branch: develop
- Git commit id: e348efa66

### JDK Version

JDK 8

### Describe the Bug

`ConsumerOffsetManager#queryMinOffsetInAllGroup(topic, filterGroups)` iterates the **live** `offsetTable.keySet()` and removes the filtered groups' entries from it:

```java
Set topicGroups = this.offsetTable.keySet(); // live view of offsetTable
if (!UtilAll.isBlank(filterGroups)) {
for (String group : filterGroups.split(",")) {
Iterator it = topicGroups.iterator();
while (it.hasNext()) {
String topicAtGroup = it.next();
if (group.equals(topicAtGroup.split(TOPIC_GROUP_SEPARATOR)[1])) {
it.remove(); // deletes the entry from offsetTable itself
removeConsumerOffset(topicAtGroup);
}
}
}
}
```

`ConcurrentHashMap.keySet()` is a live view, so `it.remove()` deletes every `topic@group` entry of the filtered groups from the real offset table. This method is called by `AdminBrokerProcessor#queryCorrectionOffset` (`RequestCode.QUERY_CORRECTION_OFFSET`), i.e. by running the read-only admin/diagnostic operation `DefaultMQAdminExt#queryCorrectionOffset(topic, compareGroup, filterGroups)`.

Consequences:

1. All consumer offsets of the filtered groups (for every topic on this broker) are wiped from memory; the next `persist()` makes the deletion permanent (`consumers.json` no longer contains the keys). With `RocksDBConsumerOffsetManager` the `removeConsumerOffset` hook deletes the rows from RocksDB immediately.
2. When the consumers of the filtered group commit/look up their offsets afterwards, `queryOffset` returns `-1` and consumption restarts according to `consumeFromWhere` — mass duplicate consumption or consumption skipping to the latest offset.
3. `topicAtGroup.split(TOPIC_GROUP_SEPARATOR)[1]` also throws `ArrayIndexOutOfBoundsException` if a malformed key without `@` is present.

This looks like a copy-paste from the real cleanup methods (`cleanOffset` / `removeOffset`): a *query* method must never mutate the table; the filter groups were only meant to be excluded from the min-offset computation.

### Steps to Reproduce

1. Start a broker, let groups `G1` and `G2` consume topic `T` so both groups have committed offsets.
2. Run the admin operation `queryCorrectionOffset(T, G1, filterGroups="G2")` once.
3. Check `consumerOffset.json` / `getConsumerStatus`: every `T@G2` offset entry is gone and gets persisted that way.

A unit test asserting `offsetTable` still contains the filtered group's entry after calling `queryMinOffsetInAllGroup` fails on current develop.

### What Did You Expect to See?

The query returns the min offsets excluding the filtered groups, without modifying `offsetTable` at all.

### What Did You See Instead?

The query deletes the filtered groups' offsets from the live table (and from RocksDB/persisted JSON), causing silent offset loss.

### Additional Context

Fix: compute on a snapshot of the key set and exclude the filter groups there, leaving `offsetTable` untouched. I will submit a PR with a regression test.

Contributor guide

Open the contributing guide

Research direction

Start in ConsumerOffsetManager#queryMinOffsetInAllGroup and trace its call from AdminBrokerProcessor#queryCorrectionOffset, then inspect the related offset-management tests. Reproduce the filtered-group query with committed offsets and verify that the min-offset result excludes those groups while offsetTable, RocksDB data, and persisted offsets remain unchanged.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
78/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.