argoproj / argoproj/notifications-engine
Enhancement Proposal: pagerdutyv2 service to support custom_details, links, and Change Events API
- Dominant language
- Go
- Stars
- 334
- Forks
- 217
- PR merge metrics
- No merged PRs in 30d
Description
## Problem
Current pagerdutyv2 service only supports 7 basic fields and cannot utilize PagerDuty's Change Events API for deployment tracking. This limits incident context and prevents PagerDuty ML from correlating deployments with incidents.
## Proposed Enhancement
Add support for:
1. `custom_details` - JSON object for rich incident context
2. `links` - Array of links for incident navigation
3. `images` - Array of images for visual context
4. `dedup_key` - Custom deduplication key
5. `timestamp` - Event timestamp
6. Change Events API endpoint (`/v2/change/enqueue`)
## Use Case
We have 100 microservices with independent on-call rotations. PagerDuty ML requires custom_details (cluster, namespace, stage, etc.) to correlate deployment changes with incidents. Current workaround uses custom webhooks.
## Webhook examples
Alert Events Template (with urgency selection):
```yaml
template.app-sync-failed: |
webhook.pagerduty-alerts: |
method: POST
body: |
{
"routing_key": "{{if eq .context.stage \"production\"}}{{if index .app.metadata.annotations \"pagerduty.io/high-urgency-key\"}}{{index .secrets (index .app.metadata.annotations \"pagerduty.io/high-urgency-key\")}}{{else}}{{.context.pagerdutyAlertsRoutingKey}}{{end}}{{else}}{{if index .app.metadata.annotations \"pagerduty.io/low-urgency-key\"}}{{index .secrets (index .app.metadata.annotations \"pagerduty.io/low-urgency-key\")}}{{else}}{{.context.pagerdutyAlertsRoutingKey}}{{end}}{{end}}",
"event_action": "trigger",
"dedup_key": "{{.app.metadata.name}}-sync-failed-{{.context.stage}}",
"payload": {
"summary": "🚨 {{.app.metadata.name}} sync failed in {{.context.stage}}",
"severity": "error",
"source": "{{.context.instance}}",
"component": "{{.app.metadata.name}}",
"group": "{{.app.spec.project}}",
"custom_details": {
"app_name": "{{.app.metadata.name}}",
"project": "{{.app.spec.project}}",
"stage": "{{.context.stage}}",
"cluster": "{{.app.spec.destination.server}}",
"namespace": "{{.app.spec.destination.namespace}}",
"error_message": "{{.app.status.operationState.message}}"
}
},
"links": [
{
"href": "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true",
"text": "View Sync Operation"
}
]
}
```
Change Events Template (to both services):
``` yaml
template.app-deployed: |
webhook.pagerduty-changes-high: |
method: POST
body: |
{
"routing_key": "{{if index .app.metadata.annotations \"pagerduty.io/high-urgency-key\"}}{{index .secrets (index .app.metadata.annotations \"pagerduty.io/high-urgency-key\")}}{{else}}{{.context.pagerdutyChangesRoutingKey}}{{end}}",
"payload": {
"summary": "🚀 {{.app.metadata.name}} deployed to {{.context.stage}}",
"timestamp": "{{.app.status.operationState.finishedAt}}",
"source": "{{.context.instance}}",
"custom_details": {
"app_name": "{{.app.metadata.name}}",
"project": "{{.app.spec.project}}",
"stage": "{{.context.stage}}",
"cluster": "{{.app.spec.destination.server}}",
"namespace": "{{.app.spec.destination.namespace}}",
"error_message": "{{.app.status.operationState.message}}"
}
}
}
```
```yaml
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: example-api
namespace: argo-cd
annotations:
# PagerDuty routing (required)
pagerduty.io/high-urgency-key: "PAGERDUTY_ACHIEVEMENTS_HIGH_KEY"
pagerduty.io/low-urgency-key: "PAGERDUTY_ACHIEVEMENTS_LOW_KEY"
# Optional: Override which events go to high urgency
# pagerduty.io/high-urgency-events: "on-sync-failed,on-deleted"
```
## References
- PagerDuty Events API v2: https://developer.pagerduty.com/docs/ZG9jOjExMDI5NTgw
- PagerDuty Change Events: https://developer.pagerduty.com/docs/ZG9jOjExMDI5NTgy
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at the existing pagerdutyv2 service and its PagerDuty Events API request path; compare the requested payload fields with the referenced Events API v2 and Change Events API documentation. Done means the service supports the listed incident fields and the Change Events API endpoint for deployment tracking.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- api, backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100