apache / apache/rocketmq-dashboard

[Studio][Bug] Cross-metric samples keep missing native alerts permanently active

Open
#3,104 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.4k
Forks
683
Avg merge
2d 14h
Merged PRs (30d)
58

Description

## Severity

Medium-High. The defect makes the native alert state and recovery-notification path untrustworthy, although it does not directly interrupt RocketMQ message traffic.

## Problem

Native alert reconciliation treats a sample from any metric as proof that every matching rule fingerprint is still present. It checks the collection scope, instance, and resource labels, but does not require the sample metric to equal the rule metric.

Because a fingerprint is derived from `ruleId + instanceId + labels`, another metric with the same labels can keep a missing metric's old state in `FIRING` or `ACKED` forever.

## Reproduction

1. Create a native business rule for `consumer.delay.seconds > 10`, scoped to instance `local` and consumer group `orders`; configure a notification channel if recovery delivery is to be observed.
2. Produce a delay sample of `120` and let the rule enter `FIRING`.
3. In the next successful full collection, keep the consumer group but make its consumption timestamp unavailable.
4. The business collector omits `consumer.delay.seconds` but still emits `consumer.lag.total` with `{consumerGroup=orders}`.
5. Process that collection.

### Actual behavior

The lag sample is added to the delay rule's `presentKeys`. The previous delay state remains active; no `RESOLVED` event or recovery notification is emitted.

### Expected behavior

Only a sample whose metric equals the rule metric may keep that rule's fingerprint present. A missing delay metric in a successful scope must resolve its prior active state even when another same-label metric remains.

## Impact

- The UI continuously reports an incident that no longer has a metric sample.
- Acknowledged alerts also remain active indefinitely.
- Operators never receive the recovery notification.
- Repeated false incidents reduce confidence in native alerting.

## Root cause

`NativeAlertProcessor.reconcileMissingActiveStates` builds `presentKeys` by applying `NativeAlertRuleScopeMatcher`, which intentionally matches instance and resource labels only. It omitted `sample.metricKey() == rule.metric`.

The real Apache business collector makes this stable: `consumer.lag.total` is always emitted for a collected group, while `consumer.delay.seconds` is conditional on a consumption timestamp being available; both use the same consumer-group label.

## Regression boundary / duplicate check

- #2679 and merged replacement PR #2957 reconcile alerts when a monitored resource disappears and no sample remains for its fingerprint.
- This case is distinct: the resource and labels still exist, but one metric disappears and another metric impersonates its presence.
- #2951/#2960 concern blank metric-name normalization, not cross-metric state reconciliation.
- #2823 scopes persisted metric snapshot history by cluster; it does not reconcile missing samples or construct `presentKeys`.
- Searches for `reconcileMissingActiveStates`, `presentKeys`, `another metric`, and `consumer.delay.seconds resolved` found no matching issue or PR.

## Suggested acceptance criteria

- Require exact metric equality while constructing reconciliation `presentKeys`.
- Preserve current behavior for a genuinely present target metric.
- Preserve the full-scope-unavailable guard against false recovery.
- Add a regression test that verifies `RESOLVED` state persistence, lifecycle event creation, original labels, and invocation of the recovery-notification enqueue path.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in NativeAlertProcessor.reconcileMissingActiveStates and inspect how NativeAlertRuleScopeMatcher contributes samples to presentKeys. Add the exact metric-match condition while preserving the full-scope-unavailable guard, then add a regression test covering RESOLVED persistence, lifecycle event creation, original labels, and recovery-notification enqueueing.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
76/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.