influxdata / influxdata/kapacitor

Retry alert on failure

Open
#1,607 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

It would be nice if kapacitor could retry alert calls on failure. A general purpose solution for all handlers would be great, but in our specific use case, we're looking for PagerDuty.

The use case is that PagerDuty (unfortunately) often fails. Normally these failures aren't a big issue as kapacitor will just perform the call again on the next point which is matches the `alert()` node's condition (or on every `StateChangesOnly` interval). However the case where this does not work is when an alert is being reset. In this case the API call is only made once and if it fails is not retried. This leaves us with orphaned alerts that we dig into, only to find the condition has cleared, just that PagerDuty wasn't updated.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by tracing alert() handling for reset notifications and the PagerDuty handler, then compare it with the existing retry behavior for repeated matching points and StateChangesOnly intervals. Done means a failed reset notification is retried so PagerDuty does not retain an orphaned alert.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.