apache / apache/rocketmq-dashboard

[Studio][Bug] Acknowledging a reminder alert never ACKs the active alert state, so reminders keep firing

Open
#4,242 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.4k
Forks
683
Avg merge
2d 14h
Merged PRs (30d)
58

Description

## Problem

Acknowledging a REMINDER system alert never ACKs the underlying alert state, so the reminder loop (and its notification outbox sends) continues after the user has acknowledged the alert. The state machine only stops reminders while the state is `ACKED`, but the acknowledge endpoint only ACKs the state for `FIRING` events.

## Evidence

- `server/src/main/java/org/apache/rocketmq/studio/ops/alert/AlertService.java` `acknowledgeAlert`: the state update is gated on `"FIRING".equalsIgnoreCase(alert.getTransition())` — a REMINDER event skips `alertStateRepository.acknowledge` entirely.
- `RmqAlertStateMapper.acknowledgeFiring` additionally requires `AND fired_at = #{firedAt}`; a REMINDER event's `time` is the reminder time (strictly after the current episode's `fired_at`), so even a widened whitelist would never satisfy the equality.
- REMINDER events are acknowledgeable in the shipped UI: `web/src/pages/ops/systemAlerts.tsx` renders the Acknowledge button for every row where `!alert.acknowledged && alert.transition !== 'RESOLVED'`, and the REMINDER row is the newest row for any alert that has been firing longer than one reminder interval (default 30m, `AlertRuleVO` default `reminderInterval`).
- Reminders are real recurring notifications: `AlertStateMachine.advanceHit` emits `AlertStateTransition.REMINDER` every reminder interval while the state row stays `FIRING`, and `NativeAlertProcessor.emitLifecycleEvent` / `NativeAlertEvaluationService` enqueue a notification for `FIRING` and `REMINDER` transitions (`emitsLifecycleEvent` includes REMINDER).

Trigger path: rule fires (state `FIRING`, `fired_at=T0`) → 30 minutes later a REMINDER event (time T1) tops the system alerts list → user clicks Acknowledge on that row → the event row is marked acknowledged, but the state row stays `FIRING` → the next evaluation emits another REMINDER and enqueues another webhook/email. Expected: acknowledgment ACKs the state and reminders stop (that is what `ACKED` exists for — `AlertStateMachine.advanceHit` returns NONE while ACKED).

## Impact

On-call operators acknowledge a firing alert and the UI marks the event acknowledged, yet paging continues indefinitely (every reminder interval) until the condition clears on its own. The acknowledge action silently fails at its only purpose for any alert that has been firing longer than one reminder interval.

## Expected behavior

Acknowledging a FIRING or REMINDER event of the current firing episode ACKs the active alert state. Events older than the current episode (a FIRING/REMINDER event from a previous, already-resolved episode) must still not ACK a newer firing state.

## Related work

- #4203 / #4204 cover the transaction boundary of the same endpoint (event write vs state write atomicity) — a different defect; this issue is about which events are allowed to ACK the state at all.
- The existing test `acknowledgingResolvedEventMustNotAcknowledgeANewerFiringStateTest` pins the stale-episode protection; any fix must preserve it.

## PR

Fix: #4243.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in server/src/main/java/org/apache/rocketmq/studio/ops/alert/AlertService.java and RmqAlertStateMapper. Read the existing acknowledgingResolvedEventMustNotAcknowledgeANewerFiringStateTest, then run the alert service tests. Done means acknowledging a current FIRING or REMINDER event stops reminders without allowing an older episode to acknowledge a newer firing state.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, typescript
Domain
api, backend, testing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.