influxdata / influxdata/kapacitor
Feature Request: "Stateless" Alerts
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
We have a use case where we'd like to alert on Docker container failures in a Mesos cluster. Since the container is never "fixed" when broken, just re-launched, there's no way to remediate a particular alert (its Mesos task ID changes upon re-launch, so the identifiers in the previous alert state would no longer match). Setting up an ``AlertNode`` for such a condition would be fine, except we'd like these container alerts not to just accumulate on the Kapacitor host, stuck in a perpetual CRITICAL state.
The ideal solution for us would be an option to make an ``AlertNode`` stateless. In other words, the alert handlers should fire when the state changes from "no state" to any other state (e.g. critical), but the alert state should not be stored on disk. This way the alert can be handled in an appropriate way (e.g. a slack message gets sent to let people know), but there's no state data accumulating on disk.
There is the ``NoRecoveries`` chain method to ``AlertNode``, but this still stores the state data on disk. So one proposal is to add an option to this method indicating whether to store the alert state (the default would be the current behavior for backwards compatibility).
Alternatively, a new chain method could be added, say ``.stateless()``.
One sticky point that comes to mind is if a user happens to mix state reset methods (``warnReset``, etc.) with ``stateless``. Methods that require access to previous state data would either need to be no-ops or error/warn in some way.
(This feature request was first discussed [here](https://community.influxdata.com/t/kapacitor-stateless-alerts/1647))
Contributor guide
Research direction
Start with the AlertNode and NoRecoveries chain method described in the issue, then review the linked community discussion for prior design context. Define how stateless alerts interact with state-reset methods and persistence; done means handlers fire on a transition from no state while no alert state accumulates on disk.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100