prometheus / prometheus/alertmanager

OpsGenie cross-team deduplication - alias prefix

Open
#3,639 5 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/enhancement
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

OpsGenie has built in deduplication of alerts.
This is a problem for us because we have multiple teams using the same OpsGenie account (different teams).
Deduplication is based on the opsGenie alias field. This field is set to the alertmanager alert key.Hash.
This means that if two teams have the same alert key, they will be deduplicated even if they are different alerts.

As a result, we have alerts which are only sent to one team, but are deduplicated on another team. The same can happen for resolving alerts. If the alert is resolved in the team that did not receive the alert, it will not be resolved in the team that did receive the alert and stay open until it is resolved manually.

One workaround is to configure a prefix for the Promtheus Integration in OpsGenie. This will add the prefix to the alias field. This will prevent deduplication between teams.
Unfortunately, this does not support updating summary and description fields (config: update_alerts).

Another workaround is to change the "group by" rules in alertmanager and add another label, e.g. team. But this configuration is error prone.

Proposed solution:
Add a new field to the OpsGenie integration configuration in alertmanager. This field will be added to the alias field. This will allow us to configure a prefix for the alias field and prevent deduplication between teams.

It will also allow use cases where the alerts from different monitoring clusters should not be deduplicated. https://github.com/prometheus/alertmanager/issues/1598

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the OpsGenie integration configuration and the code that sets the alias from the alert key hash; review how the existing update_alerts option is handled. Add the requested configurable alias prefix and verify that alerts from different teams or clusters no longer deduplicate while update behavior remains supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.