apache / apache/rocketmq

[Enhancement] Converge pop consumer offset in the background so it does not depend on client polling

Open
#10,949 3 comments 0 reactions 0 assignees View on GitHub
type/enhancement
Dominant language
Java
Stars
22.6k
Forks
12k
Avg merge
3d 1h
Merged PRs (30d)
27

Description

### Before Creating the Enhancement Request

- [x] I have confirmed that this should be classified as an enhancement rather than a bug/feature.

### Summary

With popConsumerKVServiceEnable = true, a queue's consumer offset is only advanced by pop requests, and the committed value always lags one round behind the real position.

There is no broker-side task that closes this gap, so once a consumer drains the backlog, acks everything and goes offline, the offset stays behind indefinitely and reported lag never reaches zero.

Adding background convergence would make the offset reflect the actual consumption position regardless of client polling behavior.

### Motivation

Ack no longer advances the offset (PopConsumerService#ackAsync only deletes the record), and the commit now rides on pop using the start offset of the current batch, while the real watermark nextBeginOffset goes only to pullOffsetTable.

No background task closes the gap: run() only revives, and PopConsumerCache#cleanupRecords skips the commit once the queue drains because getMinOffsetInBuffer() returns -1 — with enablePopBufferMerge=false there is no periodic committer at all.

The legacy PopBufferMergeService path does not need one, since ack itself commits nextBeginOffset.

### Describe the Solution You'd Like

Add a periodic broker-side task in PopConsumerService that converges offsetTable toward pullOffsetTable, bounded by the minimum in-flight offset still held in the cache, so it works for both enablePopBufferMerge values without depending on any further client request.

### Describe Alternatives You've Considered

/

### Additional Context

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with PopConsumerService, including run() and PopConsumerService#ackAsync, then trace how PopConsumerCache#cleanupRecords updates offsetTable and pullOffsetTable. Compare the enablePopBufferMerge paths and make the broker-side task converge offsets without client polling, while respecting the minimum in-flight cache offset.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.