prometheus / prometheus/alertmanager

Update alerts api to display that notifications are muted for certain receivers.

Open
#2,832 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/api help wanted stale
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

The is a new feature about mute_time_intervals and if approved to active_time_intervals. The notifications are not sent when they are muted but there is no way using the API what notification will be delivered or not.

Example imagine that we have an alert and we have two receivers but for devs receivers the notification is muted:

Current state:

GET /api/v2/alerts

[
  {
    "annotations": {
      "description": "randomina.lan:9100 of job node has been down for more than 10 minutes.",
      "summary": "Endpoint randomina.lan:9100 down"
    },
    "endsAt": "2022-02-14T01:07:46.140Z",
    "fingerprint": "0593f1c55738ce93",
    "receivers": [
      {
        "name": "devs"
      },
      {
        "name": "monitoring"
      }
    ],
    "startsAt": "2022-02-12T09:01:16.140Z",
    "status": {
      "inhibitedBy": [],
      "silencedBy": [],
      "state": "active"
    },
    "updatedAt": "2022-02-13T20:03:46.134-05:00",
    "generatorURL": "http://azul:9090/graph?g0.expr=up+%3D%3D+0&g0.tab=1",
    "labels": {
      "alertname": "InstanceDown",
      "instance": "randomina.lan:9100",
      "job": "node",
      "severity": "medium"
    }
  }
]

So the alert is active and is not inhibited or silenced but it wont be delivered to devs reciever so I propose the next

GET /api/v2/alerts

[
  {
    "annotations": {
      "description": "<HOST2>:9100 of job node has been down for more than 10 minutes.",
      "summary": "Endpoint <HOST2>:9100 down"
    },
    "endsAt": "<DATETIME>",
    "fingerprint": "0593f1c55738ce93",
    "receivers": [
      {
        "name": "devs"
      },
      {
        "name": "monitoring"
      }
    ],
    "startsAt": "<DATETIME>",
    "status": {
      "inhibitedBy": [],
      "notificationMutedFor": ["dev"],  <------------------------ This is new
      "silencedBy": [],
      "state": "<DATETIME>"
    },
    "updatedAt": "<DATETIME>",
    "generatorURL": "http://<HOST>:9090/graph?g0.expr=up+%3D%3D+0&g0.tab=1",
    "labels": {
      "alertname": "InstanceDown",
      "instance": "<HOST2>:9100",
      "job": "node",
      "severity": "medium"
    }
  }
]

In this case we tell that even when the alert is active the notification itself is muted for the dev reciever.

Please provide feedback if this make sense I can send a PR.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the /api/v2/alerts response and trace how mute_time_intervals and active_time_intervals are represented for alert receivers. Clarify the proposed notificationMutedFor field and its receiver values, then add API coverage showing which receivers are muted while the alert remains active. Done means the API distinguishes muted notifications from inhibited or silenced alerts.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.