prometheus / prometheus/alertmanager

Pagerduty Integration (events API V2): Updates same alert instead of creating new one

Open
#2,874 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

What did you do?
Setup Alert manager integration with Pagerduty. The configuration is setup to group by ['alertname']. I also have event orchestration setup in PD to create incident against the alert.

What did you expect to see?
Each alert with different name in alert manager should result in a separate alert (+incident) in pager duty.

What did you see instead? Under which circumstances?
For some reason, pagerduty considers every new alert as an update to existing alert and performs the update.

image
In this image, each of the updates are in fact separate alerts (different alert name). As you can see in the image below.
image

After reviewing the code (NotifyV2), I think the Dedupkey is being generated from route key, which may be the reason for issue. I only have one route with receiver setup.

The de-dup key should have been group labels.fingerprint to allow Pagerduty to identify same group updates.

&pagerDutyMessage{
		Client:      tmpl(n.conf.Client),
		ClientURL:   tmpl(n.conf.ClientURL),
		RoutingKey:  tmpl(string(n.conf.RoutingKey)),
		EventAction: eventType,
		**DedupKey:    key.Hash(),**

Environment
AlertManager, Pagerduty

  • System information:

    insert output of uname -srm here
    Darwin 19.6.0 x86_64

  • Alertmanager version:

    insert output of alertmanager --version here (repeat for each alertmanager
    version in your cluster, if relevant to the issue)
    0.23.0

  • Prometheus version:

    insert output of prometheus --version here (repeat for each prometheus
    version in your cluster, if relevant to the issue)

  • Alertmanager configuration file:


global:
  resolve_timeout: 5m 
  http_config:
      follow_redirects: true
  smtp_from: alertmanager@x.io
  smtp_hello: localhost
  smtp_smarthost: localhost:25
  smtp_require_tls: true
  pagerduty_url: https://events.pagerduty.com/v2/enqueue
  opsgenie_api_url: https://api.opsgenie.com/
  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
route:
    receiver: default-receiver
    continue: false
    routes:
        - receiver: p1
        continue: true
        group_wait: 30s
        group_interval: 30s
        repeat_interval: 4h
    receivers:
        - name: default-receiver
        email_configs:
            - send_resolved: false
            to: default@email.com
            from: alertmanager@example.org
            hello: localhost
            smarthost: localhost:25
            html: '{{ template \"email.default.html\" . }}'
            require_tls: true
        - name: p1
        pagerduty_configs:
            - send_resolved: true
            http_config:
                follow_redirects: true
            routing_key: <secret>
            url: https://events.pagerduty.com/v2/enqueue
            client: SigNoz Alert Manager
            client_url: http://localhost:8080/alerts
            description: "description"
            severity: '{{ (index .Alerts 0).Labels.severity }}'
            component: test2111}}
                
templates: []
  • Prometheus configuration file:
insert configuration here (if relevant to the issue)
  • Logs:
insert Prometheus and Alertmanager logs relevant to the issue here

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the PagerDuty NotifyV2 implementation, especially the DedupKey assignment shown in the issue. Reproduce the behavior with separate alert names and compare the generated key with the configured group labels and fingerprint. Done means distinct alert groups create separate PagerDuty incidents while updates to the same group remain deduplicated.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.