[Enhancement] Converge pop consumer offset in the background so it does not depend on client polling
- Dominant language
- Java
- Stars
- 22.6k
- Forks
- 12k
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 27
Description
### Before Creating the Enhancement Request
- [x] I have confirmed that this should be classified as an enhancement rather than a bug/feature.
### Summary
With popConsumerKVServiceEnable = true, a queue's consumer offset is only advanced by pop requests, and the committed value always lags one round behind the real position.
There is no broker-side task that closes this gap, so once a consumer drains the backlog, acks everything and goes offline, the offset stays behind indefinitely and reported lag never reaches zero.
Adding background convergence would make the offset reflect the actual consumption position regardless of client polling behavior.
### Motivation
Ack no longer advances the offset (PopConsumerService#ackAsync only deletes the record), and the commit now rides on pop using the start offset of the current batch, while the real watermark nextBeginOffset goes only to pullOffsetTable.
No background task closes the gap: run() only revives, and PopConsumerCache#cleanupRecords skips the commit once the queue drains because getMinOffsetInBuffer() returns -1 — with enablePopBufferMerge=false there is no periodic committer at all.
The legacy PopBufferMergeService path does not need one, since ack itself commits nextBeginOffset.
### Describe the Solution You'd Like
Add a periodic broker-side task in PopConsumerService that converges offsetTable toward pullOffsetTable, bounded by the minimum in-flight offset still held in the cache, so it works for both enablePopBufferMerge values without depending on any further client request.
### Describe Alternatives You've Considered
/
### Additional Context
_No response_
Contributor guide
Research direction
Start with PopConsumerService, including run() and PopConsumerService#ackAsync, then trace how PopConsumerCache#cleanupRecords updates offsetTable and pullOffsetTable. Compare the enablePopBufferMerge paths and make the broker-side task converge offsets without client polling, while respecting the minimum in-flight cache offset.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100