prometheus / prometheus/alertmanager

Webhooks for firing alerts contain zeroed EndsAt

Open
#3,351 10 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/notify kind/enhancement
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

What did you do?

It appears that webhooks for firing alerts contain zeroed EndsAt timestamps 0001-01-01T00:00:00Z instead of the actual EndsAt timestamp of the alert as can be seen in /v2/alerts.

Having looked into this further the code that causes this can be found on Line 507 of dispatch.go:

for _, alert := range alerts {
	a := *alert
	// Ensure that alerts don't resolve as time move forwards.
	if !a.ResolvedAt(now) {
		a.EndsAt = time.Time{}
	}
	alertsSlice = append(alertsSlice, &a)
}

I think the intention of this code is to prevent alerts from becoming resolved during subsequent stages of the receiver such as WaitStage and DedupStage. This can happen when a firing alert enters the stages at time t1 but doesn't finish the stages until time t2 > EndsAt.

What did you expect to see?

I expect to see either EndsAt contain a non-zero timestamp or for the EndsAt column to be omitted from the JSON for firing alerts.

What did you see instead? Under which circumstances?

Related: https://github.com/prometheus/alertmanager/issues/3341

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in dispatch/dispatch.go at line 507 and compare the alert's EndsAt there with the value returned by /v2/alerts. Read related issue #3341 to understand the intended handling during later receiver stages, then verify that firing-alert webhooks preserve a meaningful EndsAt or omit the field rather than serializing the zero timestamp.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.