influxdata / influxdata/kapacitor
If alert recovers while inhibited, it's topic will not go back to OK when it is de-inhibited
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
This could be the intended behavior, but it is unintuitive.
**OS**: Xubuntu 17.10
**Kapacitor version**: 1.5.0-1, pre-built package
**How to Reproduce:**
Using two alarms, the first one will be inhibited by the second one:
```
var data = stream
|from()
.measurement('input_data')
var trigger = data
|alert()
.category('my_category')
.warn(lambda: "time_offline" > 60)
.crit(lambda: "time_offline" > 120)
.topic('my_topic')
```
```
var data = stream
|from()
.measurement('alert_toggle')
var trigger = data
|alert()
.crit(lambda: "isOn" == 0)
.inhibit('my_category')
.topic('is_inhibited')
```
We insert a point that triggers the first alert warning level:
```shell
curl 'http://127.0.0.1:8086/write?db=XXX' --data-binary 'input_data time_offline=70
```
And the topic 'my_topic' appears:
```shell
$ kapacitor show-topic my_topic
ID: my_topic
Level: WARNING
Collected: 1
Handlers: []
Events:
Event Level Message Date
input_data:nil WARNING input_data:nil is WARNING 05 Jul 18 20:38 -03
```
Now, we inhibit the alarm:
```shell
$ curl 'http://127.0.0.1:8086/write?db=XXX' --data-binary 'alert_toggle isOn=0'
$ kapacitor show-topic is_inhibited
ID: is_inhibited
Level: CRITICAL
Collected: 1
Handlers: []
Events:
Event Level Message Date
alert_toggle:nil CRITICAL alert_toggle:nil is CRITICAL 05 Jul 18 20:41 -03
```
Now we reset the first alert, but it does not, as it is inhibited.
```shell
$ curl 'http://127.0.0.1:8086/write?db=XXX' --data-binary 'input_data time_offline=30'
$ kapacitor show-topic my_topic
ID: my_topic
Level: WARNING
Collected: 1
Handlers: []
Events:
Event Level Message Date
input_data:nil WARNING input_data:nil is WARNING 05 Jul 18 20:38 -03
```
Now, here is what bugs me.
We de-inhibit the alert, and try to reset it again.
```shell
$ curl 'http://127.0.0.1:8086/write?db=XXX' --data-binary 'alert_toggle isOn=1'
$ kapacitor show-topic is_inhibited
ID: is_inhibited
Level: OK
Collected: 2
Handlers: []
Events:
Event Level Message Date
alert_toggle:nil OK alert_toggle:nil is OK 05 Jul 18 20:43 -03
$ curl 'http://127.0.0.1:8086/write?db=XXX' --data-binary 'input_data time_offline=30'
$ kapacitor show-topic my_topic
ID: my_topic
Level: WARNING
Collected: 1
Handlers: []
Events:
Event Level Message Date
input_data:nil WARNING input_data:nil is WARNING 05 Jul 18 20:38 -03
```
Now the alert can not be reset, until it changes its state either to INFO, WARN, or CRIT, and goes back again.
If we add an httpOut() endpoint to the alert, we can see it has correctly switched to OK status, but the topic will not.
**What was expected**
The inhibited alert should have updated its topic the next time it received data after being de-inhibited.
It appears that the alert internal status is still updated while inhibited, even if no notification is sent. This causes the status change event to be lost when the alert is enabled again.
Contributor guide
Assessment
This issue has not been assessed yet.