influxdata / influxdata/kapacitor
Alerts suddenly start getting inhibited
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
This evening, the alert node suddenly started inhibiting alerts even there is no .inhibit() property defined. Events from influxdb continued to be received by kapacitor.
Tick Script Template:
```
var measurement string
var where_filter = lambda: TRUE
var groups = [*]
var window = 10s
var deadman_interval = 900s
var topic string
var info = lambda: TRUE
var warn = lambda: FALSE
var crit = lambda: FALSE
var data_raw = stream
|from()
.measurement(measurement)
.where(where_filter)
.groupBy(groups)
data_raw
|alert()
.category('threshold')
.info(info)
.warn(warn)
.crit(crit)
.topic(topic)
[...]
```
Variables:
```
Vars:
Name Type Value
crit lambda "used_percent" > 90.0
measurement string disk
topic string telegraf_disk
warn lambda "used_percent" > 75.0
```
Graph:
```
digraph disk {
graph [throughput="20.01 points/s"];
stream0 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
stream0 -> from1 [processed="111222"];
from1 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
from1 -> alert2 [processed="111222"];
alert2 [alerts_inhibited="18148" alerts_triggered="93074" avg_exec_time_ns="0s" crits_triggered="0" errors="0" infos_triggered="92811" oks_triggered="0" warns_triggered="263" working_cardinality="352" ];
```
Version:
kapacitor-1.6.5-1.x86_64 on Centos 7, Influxdb is 1.8.10
Data is gathered from `telegraf` and sent to influxdb; kapacitor subscribed to the DB. Any pointers?
Contributor guide
Research direction
Start with the alert() node in the provided TICK script and reproduce the graph showing alerts_inhibited without an inhibit property. Inspect the alert node's inhibition metrics and behavior using the supplied template, variables, and Kapacitor 1.6.5 environment; done means identifying the cause and confirming that alerts are no longer inhibited unexpectedly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100