linkedin / linkedin/iris

Feature request: Extract fields from AlertManager labels into top level context

Open
#704 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
845
Forks
146
PR merge metrics
No merged PRs in 30d

Description

Hey folks, I'm liking Iris so far. I'd like to check receptiveness to this change - I'm happy to put up a PR for this, as it would help us.

### Problem

The AlertManager integration doesn't currently support the "override incident title" and "plan dynamic targets" features.

AlertManager sends a payload in this (slightly redacted) format:

```
{
"groupKey": "Demo system down",
"status": "firing",
"groupLabels": {"iris_plan": "page-engineer", "severity":"critical", "service":"demo-system"},
"commonLabels": {"iris_plan": "page-engineer", "severity":"critical", "service":"demo-system"},
"commonAnnotations": {"team":"demo-team", "runbook_url": "https://example.org", "description": "The service is down!"}
}
```

Since the `groupLabels.iris_plan` is set to an existing plan (`page-engineer`) the plan will run and the `context` (the above payload) will be available in the incident summary and context templates and the message templates.

Unfortunately, if we want to react to all incidents in the same way (i.e. page an on-call primary and, if they don't answer, the secondary) but want to pages to be received by different teams, we must create a plan (or potentially plans) for each team, and set the `iris_plan` label on each alert to be team specific (i.e. `iris_plan: "page-engineer-demo-team"`. Ideally we'd be able to use the existing dynamic_targets feature on plans so that we only need to create one plan which could be reused by all teams.

Also, each incident has the title "alertmanager" (the name of the application). Though we can set an `

{{groupLabels.alertname}}

` in the context templates, it would be nice if this feature was supported for the AlertManager integration.

### Feature request

In the [AlertManager integration](https://github.com/linkedin/iris/blob/master/src/iris/webhooks/alertmanager.py#L34-L35) it'd be great if we could pull up the `alertname` field from `groupLabels` if it exists, and allow users to set dynamic targets in alert labels.

```yaml
labels:
alertname: "Demo service is down"
iris_plan: page-engineer
iris_targets: '[{"role": "oncall-primary","target": "demo-team"},{"role": "oncall-secondary","target": "demo-team"}]'
```

Contributor guide

Open the contributing guide

Research direction

Start in src/iris/webhooks/alertmanager.py around the integration code linked in the issue, and trace how the AlertManager payload becomes the incident title, context, and plan inputs. Check the existing dynamic_targets handling and related tests before deciding how groupLabels.alertname and iris_targets should be consumed. Done means AlertManager labels can provide the incident title and dynamic targets without requiring team-specific plans.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
api, backend
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.