influxdata / influxdata/kapacitor

If alert recovers while inhibited, it's topic will not go back to OK when it is de-inhibited

Open
#1,995 2 comments 0 reactions 1 assignee Claimed by @docmerlin View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

This could be the intended behavior, but it is unintuitive.

**OS**: Xubuntu 17.10
**Kapacitor version**: 1.5.0-1, pre-built package

**How to Reproduce:**

Using two alarms, the first one will be inhibited by the second one:

```
var data = stream
|from()
.measurement('input_data')
var trigger = data
|alert()
.category('my_category')
.warn(lambda: "time_offline" > 60)
.crit(lambda: "time_offline" > 120)
.topic('my_topic')
```

```
var data = stream
|from()
.measurement('alert_toggle')
var trigger = data
|alert()
.crit(lambda: "isOn" == 0)
.inhibit('my_category')
.topic('is_inhibited')
```
We insert a point that triggers the first alert warning level:

```shell
curl 'http://127.0.0.1:8086/write?db=XXX' --data-binary 'input_data time_offline=70
```
And the topic 'my_topic' appears:
```shell
$ kapacitor show-topic my_topic
ID: my_topic
Level: WARNING
Collected: 1
Handlers: []
Events:
Event Level Message Date
input_data:nil WARNING input_data:nil is WARNING 05 Jul 18 20:38 -03
```

Now, we inhibit the alarm:

```shell
$ curl 'http://127.0.0.1:8086/write?db=XXX' --data-binary 'alert_toggle isOn=0'
$ kapacitor show-topic is_inhibited
ID: is_inhibited
Level: CRITICAL
Collected: 1
Handlers: []
Events:
Event Level Message Date
alert_toggle:nil CRITICAL alert_toggle:nil is CRITICAL 05 Jul 18 20:41 -03
```
Now we reset the first alert, but it does not, as it is inhibited.

```shell
$ curl 'http://127.0.0.1:8086/write?db=XXX' --data-binary 'input_data time_offline=30'
$ kapacitor show-topic my_topic
ID: my_topic
Level: WARNING
Collected: 1
Handlers: []
Events:
Event Level Message Date
input_data:nil WARNING input_data:nil is WARNING 05 Jul 18 20:38 -03
```
Now, here is what bugs me.
We de-inhibit the alert, and try to reset it again.

```shell
$ curl 'http://127.0.0.1:8086/write?db=XXX' --data-binary 'alert_toggle isOn=1'
$ kapacitor show-topic is_inhibited
ID: is_inhibited
Level: OK
Collected: 2
Handlers: []
Events:
Event Level Message Date
alert_toggle:nil OK alert_toggle:nil is OK 05 Jul 18 20:43 -03

$ curl 'http://127.0.0.1:8086/write?db=XXX' --data-binary 'input_data time_offline=30'
$ kapacitor show-topic my_topic
ID: my_topic
Level: WARNING
Collected: 1
Handlers: []
Events:
Event Level Message Date
input_data:nil WARNING input_data:nil is WARNING 05 Jul 18 20:38 -03
```
Now the alert can not be reset, until it changes its state either to INFO, WARN, or CRIT, and goes back again.

If we add an httpOut() endpoint to the alert, we can see it has correctly switched to OK status, but the topic will not.

**What was expected**

The inhibited alert should have updated its topic the next time it received data after being de-inhibited.

It appears that the alert internal status is still updated while inhibited, even if no notification is sent. This causes the status change event to be lost when the alert is enabled again.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.