influxdata / influxdata/kapacitor
clear alert when data stops coming in
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
We are writing alerts based on data from telegraf's procstat input (such as when `num_fds` is approaching `rlimit_num_fds_soft`). However since processes can disappear at any time, we need a way to reset any active alert when we stop receiving data for it. We have telegraf set to collect data every 15 seconds, so we would like to do a very conservative silence period of 60s after which if no data is received for a given pid, any active alert is reset to OK state. To be even safer it would be nice to reset it only if we are still receiving data from the host, but this would likely make the script overly complex and not justified.
My first thought on accomplishing this was to use a `stats()` node to clear the alert condition once we go 60s without any incoming data. However this requires joining the stats output onto the input data, which is currently not supported (ref #1334).
As such I cannot see any way to currently accomplish this goal.
I'm also familiar with the kapacitor code base, and I can also think of a related issue to this. If we group by pid, this is going to result in unbounded memory growth within kapacitor (as more and more pids come in), as many nodes permanently track each group they receive, and never expire them.
Contributor guide
Research direction
The issue names no files, tests, or entry points; start by locating Kapacitor's alert-state handling and the lifecycle of grouped data. Done means an active alert for a process is reset after 60 seconds without data, while the design also addresses unbounded tracking of groups such as pids.
Written by the indexing model from the issue text.
Assessment
- Domain
- observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100