argoproj / argoproj/notifications-engine

Enhancement Proposal: pagerdutyv2 service to support custom_details, links, and Change Events API

Open
#409 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
334
Forks
217
PR merge metrics
No merged PRs in 30d

Description

## Problem
Current pagerdutyv2 service only supports 7 basic fields and cannot utilize PagerDuty's Change Events API for deployment tracking. This limits incident context and prevents PagerDuty ML from correlating deployments with incidents.

## Proposed Enhancement
Add support for:
1. `custom_details` - JSON object for rich incident context
2. `links` - Array of links for incident navigation
3. `images` - Array of images for visual context
4. `dedup_key` - Custom deduplication key
5. `timestamp` - Event timestamp
6. Change Events API endpoint (`/v2/change/enqueue`)

## Use Case
We have 100 microservices with independent on-call rotations. PagerDuty ML requires custom_details (cluster, namespace, stage, etc.) to correlate deployment changes with incidents. Current workaround uses custom webhooks.

## Webhook examples

Alert Events Template (with urgency selection):

```yaml
template.app-sync-failed: |
webhook.pagerduty-alerts: |
method: POST
body: |
{
"routing_key": "{{if eq .context.stage \"production\"}}{{if index .app.metadata.annotations \"pagerduty.io/high-urgency-key\"}}{{index .secrets (index .app.metadata.annotations \"pagerduty.io/high-urgency-key\")}}{{else}}{{.context.pagerdutyAlertsRoutingKey}}{{end}}{{else}}{{if index .app.metadata.annotations \"pagerduty.io/low-urgency-key\"}}{{index .secrets (index .app.metadata.annotations \"pagerduty.io/low-urgency-key\")}}{{else}}{{.context.pagerdutyAlertsRoutingKey}}{{end}}{{end}}",
"event_action": "trigger",
"dedup_key": "{{.app.metadata.name}}-sync-failed-{{.context.stage}}",
"payload": {
"summary": "🚨 {{.app.metadata.name}} sync failed in {{.context.stage}}",
"severity": "error",
"source": "{{.context.instance}}",
"component": "{{.app.metadata.name}}",
"group": "{{.app.spec.project}}",
"custom_details": {
"app_name": "{{.app.metadata.name}}",
"project": "{{.app.spec.project}}",
"stage": "{{.context.stage}}",
"cluster": "{{.app.spec.destination.server}}",
"namespace": "{{.app.spec.destination.namespace}}",
"error_message": "{{.app.status.operationState.message}}"
}
},
"links": [
{
"href": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true",
"text": "View Sync Operation"
}
]
}
```

Change Events Template (to both services):

``` yaml
template.app-deployed: |
webhook.pagerduty-changes-high: |
method: POST
body: |
{
"routing_key": "{{if index .app.metadata.annotations \"pagerduty.io/high-urgency-key\"}}{{index .secrets (index .app.metadata.annotations \"pagerduty.io/high-urgency-key\")}}{{else}}{{.context.pagerdutyChangesRoutingKey}}{{end}}",
"payload": {
"summary": "🚀 {{.app.metadata.name}} deployed to {{.context.stage}}",
"timestamp": "{{.app.status.operationState.finishedAt}}",
"source": "{{.context.instance}}",
"custom_details": {
"app_name": "{{.app.metadata.name}}",
"project": "{{.app.spec.project}}",
"stage": "{{.context.stage}}",
"cluster": "{{.app.spec.destination.server}}",
"namespace": "{{.app.spec.destination.namespace}}",
"error_message": "{{.app.status.operationState.message}}"
}
}
}
```

```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: example-api
namespace: argo-cd
annotations:
# PagerDuty routing (required)
pagerduty.io/high-urgency-key: "PAGERDUTY_ACHIEVEMENTS_HIGH_KEY"
pagerduty.io/low-urgency-key: "PAGERDUTY_ACHIEVEMENTS_LOW_KEY"

# Optional: Override which events go to high urgency
# pagerduty.io/high-urgency-events: "on-sync-failed,on-deleted"
```

## References
- PagerDuty Events API v2: https://developer.pagerduty.com/docs/ZG9jOjExMDI5NTgw
- PagerDuty Change Events: https://developer.pagerduty.com/docs/ZG9jOjExMDI5NTgy

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the existing pagerdutyv2 service and its PagerDuty Events API request path; compare the requested payload fields with the referenced Events API v2 and Change Events API documentation. Done means the service supports the listed incident fields and the Change Events API endpoint for deployment tracking.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
api, backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.