influxdata / influxdata/kapacitor

clear alert when data stops coming in

Open
#1,680 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

We are writing alerts based on data from telegraf's procstat input (such as when `num_fds` is approaching `rlimit_num_fds_soft`). However since processes can disappear at any time, we need a way to reset any active alert when we stop receiving data for it. We have telegraf set to collect data every 15 seconds, so we would like to do a very conservative silence period of 60s after which if no data is received for a given pid, any active alert is reset to OK state. To be even safer it would be nice to reset it only if we are still receiving data from the host, but this would likely make the script overly complex and not justified.

My first thought on accomplishing this was to use a `stats()` node to clear the alert condition once we go 60s without any incoming data. However this requires joining the stats output onto the input data, which is currently not supported (ref #1334).
As such I cannot see any way to currently accomplish this goal.

I'm also familiar with the kapacitor code base, and I can also think of a related issue to this. If we group by pid, this is going to result in unbounded memory growth within kapacitor (as more and more pids come in), as many nodes permanently track each group they receive, and never expire them.

Contributor guide

Open the contributing guide

Research direction

The issue names no files, tests, or entry points; start by locating Kapacitor's alert-state handling and the lifecycle of grouped data. Done means an active alert for a process is reset after 60 seconds without data, while the design also addresses unbounded tracking of groups such as pids.

Written by the indexing model from the issue text.

Assessment

Domain
observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.