hyperdxio / hyperdxio/hyperdx

Alerts re-notify on every evaluation while firing

Open
#2,464 1 comment 7 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Backend
Dominant language
TypeScript
Stars
9.9k
Forks
471
Avg merge
2d 4h
Merged PRs (30d)
117

Description

A firing alert sends a notification on every evaluation interval for as long as the condition persists (e.g. every 5m).
For a sustained condition like high node memory, this produces a continuous stream of duplicate notifications until it resolves.
There's no setting to notify only on the OK→ALERT transition, nor a re-notification interval / cooldown.
This option would reduce massively the amount of alert we are receiving at the moment.

Current behavior (packages/api/src/tasks/checkAlerts/index.ts, HyperDX 2.27.0)
The ALERT notification is sent unconditionally whenever the threshold is exceeded:

if (doesExceedThreshold(alert, value)) {
  history.state = AlertState.ALERT;
  history.counts += 1;
  await trySendNotification({ state: AlertState.ALERT, ... });
}

By contrast, the resolve notification is edge-gated, firing only on the transition:

if (previousHistory?.state === AlertState.ALERT && currentHistory.state === AlertState.OK) {
  await trySendNotification({ state: AlertState.OK, ... });
}

Version: HyperDX 2.27.0

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in packages/api/src/tasks/checkAlerts/index.ts, where ALERT notifications are sent on every evaluation and OK notifications are transition-gated. Trace the alert history and notification flow to define the OK→ALERT behavior and re-notification or cooldown option; done means sustained alerts no longer produce duplicate notifications on every interval while resolution still notifies.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.