influxdata / influxdata/kapacitor

Alerts suddenly start getting inhibited

Open
#2,773 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

This evening, the alert node suddenly started inhibiting alerts even there is no .inhibit() property defined. Events from influxdb continued to be received by kapacitor.

Tick Script Template:
```
var measurement string

var where_filter = lambda: TRUE

var groups = [*]

var window = 10s

var deadman_interval = 900s

var topic string

var info = lambda: TRUE

var warn = lambda: FALSE

var crit = lambda: FALSE

var data_raw = stream
|from()
.measurement(measurement)
.where(where_filter)
.groupBy(groups)

data_raw
|alert()
.category('threshold')
.info(info)
.warn(warn)
.crit(crit)
.topic(topic)
[...]
```

Variables:
```
Vars:
Name Type Value
crit lambda "used_percent" > 90.0
measurement string disk
topic string telegraf_disk
warn lambda "used_percent" > 75.0
```

Graph:
```
digraph disk {
graph [throughput="20.01 points/s"];

stream0 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
stream0 -> from1 [processed="111222"];

from1 [avg_exec_time_ns="0s" errors="0" working_cardinality="0" ];
from1 -> alert2 [processed="111222"];

alert2 [alerts_inhibited="18148" alerts_triggered="93074" avg_exec_time_ns="0s" crits_triggered="0" errors="0" infos_triggered="92811" oks_triggered="0" warns_triggered="263" working_cardinality="352" ];

```
Version:
kapacitor-1.6.5-1.x86_64 on Centos 7, Influxdb is 1.8.10

Data is gathered from `telegraf` and sent to influxdb; kapacitor subscribed to the DB. Any pointers?

Contributor guide

Open the contributing guide

Research direction

Start with the alert() node in the provided TICK script and reproduce the graph showing alerts_inhibited without an inhibit property. Inspect the alert node's inhibition metrics and behavior using the supplied template, variables, and Kapacitor 1.6.5 environment; done means identifying the cause and confirming that alerts are no longer inhibited unexpectedly.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.