[Bug] Deleted LiteTopics remain in lag latency metrics and timestamp TopK
- Dominant language
- Java
- Stars
- 22.6k
- Forks
- 12k
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 27
Description
### Runtime platform environment
Windows, local broker unit tests with mocked queue state; no running cluster is required for the regression.
### RocketMQ version
`develop` at `ff8f6f74c560e391261ccd716707c6d20422e253` (5.5.1).
### JDK Version
Amazon Corretto 8u482; Maven 3.9.11.
### Describe the Bug
Lite lag timestamps are cached by `LiteConsumerLagCalculator`. TTL cleanup deletes a LiteTopic's queue, offsets, and subscriptions, but its cached lag timestamp can remain. Both `calculateLiteLagLatency` and `getLagTimestampTopK` continue to use that timestamp without checking whether the queue exists.
An expired session can therefore remain the oldest entry in a group's latency metric or occupy a TopK slot. With no further POP requests for that session, the normal POP-driven cache removal need not run.
### Steps to Reproduce
1. Record a lag timestamp for a LiteTopic through `updateLagInfo`.
2. Delete its queue, as TTL cleanup does. The lifecycle manager now returns a non-positive maximum offset for that LMQ.
3. Call `getLagTimestampTopK`, or enable `liteLagLatencyMetricsEnable` and call `calculateLiteLagLatency`.
The accompanying unit test models the queue becoming absent and checks the public calculator results. On the unmodified implementation, the assertion that the expired topic is absent from TopK fails.
### What Did You Expect to See?
Expired queues should be excluded from latency metrics and timestamp TopK results, and their stale samples should be removed from the cache. Samples for other queues should be preserved. A group whose cached samples are all expired should produce no latency observation and an empty timestamp TopK result.
### What Did You See Instead?
The cache continues to supply a timestamp for the deleted queue. Aggregate latency can keep increasing, and timestamp TopK can select that entry even though its queue is gone.
### Additional Context
A deletion-only cleanup has a race: a POP request may publish a timestamp that it read before deletion after that cleanup has finished. Validating cached samples when reporting or querying also handles such late updates while the queue remains absent.
The proposed fix uses the existing lifecycle manager's in-memory maximum-offset lookup for cached candidates. When removing a stale sample, identity matching is needed because `LagTimeInfo.equals` compares only the LMQ name; removing by equality could delete a concurrent replacement for a recreated session.
This is cleanup at metric/query time, not an atomic snapshot with queue deletion. A collection overlapping deletion can observe the earlier state; a subsequent collection revalidates it. No live-cluster test was run. Existing related work such as #10723 concerns deciding TTL expiry, rather than stale lag samples after deletion.
Contributor guide
Research direction
Start by reading LiteConsumerLagCalculator, especially calculateLiteLagLatency, getLagTimestampTopK, and updateLagInfo, then inspect the lifecycle manager's maximum-offset lookup used to detect absent queues. Run the accompanying unit test that models a queue becoming absent and checks the public calculator results. Done means expired samples are removed without removing a concurrent replacement, other queues remain, and all-expired results contain no observation or TopK entries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 69/100