apache / apache/rocketmq

[Bug] PopReviveService may stuck on check if many messages change invisible in the same time.

Open
#9,956 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
22.6k
Forks
12k
Avg merge
2d 20h
Merged PRs (30d)
26

Description

### Before Creating the Bug Report

- [x] I found a bug, not just asking a question, which should be created in [GitHub Discussions](https://github.com/apache/rocketmq/discussions).

- [x] I have searched the [GitHub Issues](https://github.com/apache/rocketmq/issues) and [GitHub Discussions](https://github.com/apache/rocketmq/discussions) of this repository and believe that this is not a duplicate.

- [x] I have confirmed that this bug belongs to the current repository, not other repositories of RocketMQ.

### Runtime platform environment

Any

### RocketMQ version

5.x

### JDK Version

Any

### Describe the Bug

```
"PopReviveService_2" Id=121 cpuUsage=2.93% deltaTime=11ms time=603096ms RUNNABLE
at com.alibaba.fastjson.parser.deserializer.FieldDeserializer.setValue(FieldDeserializer.java:167)
at com.alibaba.fastjson.parser.deserializer.DefaultFieldDeserializer.parseField(DefaultFieldDeserializer.java:126)
at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.parseField(JavaBeanDeserializer.java:1278)
at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.deserialze(JavaBeanDeserializer.java:893)
at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.parseRest(JavaBeanDeserializer.java:1624)
at com.alibaba.fastjson.parser.deserializer.FastjsonASMDeserializer_16_PopCheckPoint.deserialze(Unknown Source)
at com.alibaba.fastjson.parser.deserializer.JavaBeanDeserializer.deserialze(JavaBeanDeserializer.java:287)
at com.alibaba.fastjson.parser.DefaultJSONParser.parseObject(DefaultJSONParser.java:705)
at com.alibaba.fastjson.JSON.parseObject(JSON.java:394)
at com.alibaba.fastjson.JSON.parseObject(JSON.java:298)
at com.alibaba.fastjson.JSON.parseObject(JSON.java:588)
at org.apache.rocketmq.broker.processor.PopReviveService.consumeReviveMessage(PopReviveService.java:366)
at org.apache.rocketmq.broker.processor.PopReviveService.run(PopReviveService.java:654)
at java.lang.Thread.run(Thread.java:748)

"PopReviveService_0" Id=119 cpuUsage=2.82% deltaTime=11ms time=668229ms RUNNABLE
at io.opentelemetry.api.internal.ImmutableKeyValuePairs.splitAndMerge(ImmutableKeyValuePairs.java:143)
at io.opentelemetry.api.internal.ImmutableKeyValuePairs.splitAndMerge(ImmutableKeyValuePairs.java:143)
at io.opentelemetry.api.internal.ImmutableKeyValuePairs.mergeSort(ImmutableKeyValuePairs.java:119)
at io.opentelemetry.api.internal.ImmutableKeyValuePairs.sortAndFilter(ImmutableKeyValuePairs.java:110)
at io.opentelemetry.api.internal.ImmutableKeyValuePairs.(ImmutableKeyValuePairs.java:51)
at io.opentelemetry.api.common.ArrayBackedAttributes.(ArrayBackedAttributes.java:26)
at io.opentelemetry.api.common.ArrayBackedAttributes.sortAndFilterToAttributes(ArrayBackedAttributes.java:60)
at io.opentelemetry.api.common.ArrayBackedAttributesBuilder.build(ArrayBackedAttributesBuilder.java:31)
at org.apache.rocketmq.broker.metrics.PopMetricsManager.incPopReviveGetCount(PopMetricsManager.java:304)
at org.apache.rocketmq.broker.metrics.PopMetricsManager.incPopReviveCkGetCount(PopMetricsManager.java:293)
at org.apache.rocketmq.broker.processor.PopReviveService.consumeReviveMessage(PopReviveService.java:371)
at org.apache.rocketmq.broker.processor.PopReviveService.run(PopReviveService.java:654)
at java.lang.Thread.run(Thread.java:748)
```

the PopReviveService keep trying consumeReviveMessage and mergeAndRevive, but the logs showed the newOffset did not changed for a long time.

```
2025-12-23 14:51:04.631 DEBUG PopReviveService_2 - reviveQueueId=2, scan timeout
2025-12-23 14:51:04.834 DEBUG PopReviveService_5 - reviveQueueId=5, scan timeout
2025-12-23 14:51:04.937 DEBUG PopReviveService_6 - reviveQueueId=6, scan timeout
2025-12-23 14:51:05.031 DEBUG PopReviveService_3 - reviveQueueId=3, scan timeout
2025-12-23 14:51:05.031 DEBUG PopReviveService_1 - reviveQueueId=1, scan timeout
2025-12-23 14:51:05.031 DEBUG PopReviveService_0 - reviveQueueId=0, scan timeout
2025-12-23 14:51:05.031 DEBUG PopReviveService_7 - reviveQueueId=7, scan timeout
2025-12-23 14:51:05.031 DEBUG PopReviveService_4 - reviveQueueId=4, scan timeout
2025-12-23 14:51:05.731 DEBUG PopReviveService_2 - reviveQueueId=2, ck listSize=35742
2025-12-23 14:51:05.731 DEBUG PopReviveService_2 - reviveQueueId=2, 1st ck, startOffset=9131128, reviveOffset=1822555; last ck, startOffset=8831568, reviveOffset=1859338
2025-12-23 14:51:05.731 DEBUG PopReviveService_2 - reviveQueueId=2, revive finish,old offset is 1822554, new offset is 1822554, ckDelay=12499
2025-12-23 14:51:05.835 DEBUG PopReviveService_6 - reviveQueueId=6, ck listSize=33396
2025-12-23 14:51:05.835 DEBUG PopReviveService_6 - reviveQueueId=6, 1st ck, startOffset=2809176, reviveOffset=1821380; last ck, startOffset=8825699, reviveOffset=1855715
2025-12-23 14:51:05.835 DEBUG PopReviveService_6 - reviveQueueId=6, revive finish,old offset is 1821379, new offset is 1821379, ckDelay=12499
2025-12-23 14:51:05.953 INFO PopLongPollingService - pollingMapSize=1,tmpTotalSize=64,atomicTotalSize=64,diffSize=0
2025-12-23 14:51:06.031 DEBUG PopReviveService_4 - reviveQueueId=4, ck listSize=37939
2025-12-23 14:51:06.031 DEBUG PopReviveService_4 - reviveQueueId=4, 1st ck, startOffset=9131160, reviveOffset=1817744; last ck, startOffset=8836843, reviveOffset=1856511
2025-12-23 14:51:06.031 DEBUG PopReviveService_4 - reviveQueueId=4, revive finish,old offset is 1817743, new offset is 1817743, ckDelay=12499
```

### Steps to Reproduce

1. using 5x
2. use pop consumer, and change invisible for 10s for every messages
3. start more than 500 consumers for the same group
4. keep running for hours to make revive log long enough.
### What Did You Expect to See?

the revive log consume offset should be updated

### What Did You See Instead?

the revive log consume offset did not update for a long time.

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start in PopReviveService.consumeReviveMessage at lines 366-371 and its run loop at line 654, then trace the mergeAndRevive path while reviewing the logged offset values. Reproduce with pop consumers changing visibility for 10 seconds and more than 500 consumers in one group. Done means the revive log consume offset advances instead of remaining unchanged for a long time.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.