influxdata / influxdata/kapacitor
Retry alert on failure
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
It would be nice if kapacitor could retry alert calls on failure. A general purpose solution for all handlers would be great, but in our specific use case, we're looking for PagerDuty.
The use case is that PagerDuty (unfortunately) often fails. Normally these failures aren't a big issue as kapacitor will just perform the call again on the next point which is matches the `alert()` node's condition (or on every `StateChangesOnly` interval). However the case where this does not work is when an alert is being reset. In this case the API call is only made once and if it fails is not retried. This leaves us with orphaned alerts that we dig into, only to find the condition has cleared, just that PagerDuty wasn't updated.
Contributor guide
Research direction
No files or tests are named. Start by tracing alert() handling for reset notifications and the PagerDuty handler, then compare it with the existing retry behavior for repeated matching points and StateChangesOnly intervals. Done means a failed reset notification is retried so PagerDuty does not retain an orphaned alert.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100