prometheus / prometheus/alertmanager

Add receiver/integration (and where available, channel) as structured log fields on notification failures

Open
#5,396 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

What

When a notification send fails, the dispatcher logs:

level=ERROR source=dispatch.go msg="Notify for alerts failed" component=dispatcher aggrGroup="..." num_alerts=1 err="<receiver>/<integration>[0]: notify retry canceled due to unrecoverable error after 1 attempts: channel \"CXXXXXXXX\": unexpected status code 429: {\"ok\":false,\"error\":\"ratelimited\"}"

The receiver name, integration type, and (for integrations that surface it, e.g. Slack) the target channel are all present — but only concatenated into the free-text err string. The only structured fields are component, aggrGroup, num_alerts, and err.

Why

Operators aggregating these logs (Loki, ELK, etc.) can't group or alert by receiver / integration / channel without regex-parsing err, which is brittle across error shapes.

This matters specifically for Slack rate-limiting: Slack's chat.postMessage limit is 1 message/sec per channel, so 429s are inherently channel-scoped. Diagnosing "which channel / receiver is being rate-limited" is the exact question operators need to answer, and today it requires parsing the error string. A structured receiver (and, where the notifier exposes it, channel) field would make this a first-class group-by.

The RetryStage already holds the integration and receiver name at the point of failure (in notify.go, where the stage is constructed via NewRetryStage(integration, groupName, …)), so receiver and integration are available to attach to the logger with no new plumbing. Channel is integration-specific and lives inside the wrapped notifier error, so it's a softer ask — but receiver/integration alone would cover most of the need.

Proposal

Attach receiver and integration as structured fields on the failure log in dispatch.go (and/or notify.go's retry logging), e.g.:

logger := d.logger.With("aggrGroup", ag.GroupKey(), "num_alerts", len(alerts), "receiver", ..., "integration", ..., "err", err)

Optionally, integrations that already extract a channel/target could surface it as a structured field on their own error path.

Happy to open a PR if the maintainers are open to the approach.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Read dispatch.go and notify.go, starting at the failure logging and the NewRetryStage(integration, groupName, …) construction. Trace how receiver and integration are available at the failure point, and inspect whether any notifier errors expose a channel. Done means notification failures include receiver and integration as structured fields, with channel surfaced only where already available.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.