influxdata / influxdata/kapacitor

Only 1 event is fired after reload task forcing to change to OK status N active events

Open
#2,061 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Hi,

We have been working with Kapacitor to generate alerts based on metrics threshold (simple ones) on:

__SO__: RHEL 7.4
__Kapacitor__: Kapacitor OSS 1.5.0 (git: HEAD 4f10efc41b4dcac070495cf95ba2c41cfcc2aa3a)

## Overview

We have some TICKScripts that fires `N` events, based on `working cardinality` of the `alert` node, so the N events can be changing his own `state` based on the threshold.

The problem seems to appear when we change the TICKScript and we reload the task, forcing the __OK__ of the `N events`

### Actual behaviour

After reload the task with new thresholds to force the __OK__ on the `N events`, only 1 event is fired to __OK__ and the other `N-1 events` seems to be 'lost' and considered as __OK__, but no __OK__ event is fired.

### Expected behaviour

After reload the task with new thresholds to force the __OK__ on the `N events`, the `N events` are fired to __OK__.

## Detailed case

To allow you to repro the case, I have written a TICKScript and a brief table with actions and events fired:

### TICKSCRIPT
````
var ID = 'ticks_cpu'
var FIELD = 'usage-idle'
var FIELD_DEFAULT = 0.0
var TH_CRIT_DEF = 0.0
var TH_WARN_DEF = 0.0
var TH_INFO_DEF = 0.0

// TICKSCRIPT:
// ================
// var data = stream
stream
|from()
.database('telegraf')
.retentionPolicy('autogen')
.measurement('cpu')
.groupBy(*)
|default()
.field(FIELD, FIELD_DEFAULT)
|eval()
.keep(FIELD)
|window()
.period(1m)
.every(10s)
.align()
|mean(FIELD)
.as('value')
|alert()
.crit(lambda: float("value") < TH_CRIT_DEF)
.warn(lambda: float("value") < TH_WARN_DEF)
.info(lambda: float("value") < TH_INFO_DEF)
.id(ID)
.log('/tmp/test-cpu.log')
````

### Actions and results

On the following table, it is shown the actions and the events results.

As it is shown, after forcing an __OK__ on already `N CRIT events`, it only fires a single __OK__ event

Step | Action | #Cores | #Actual Events | Expected result | Example
-- | -- | -- | -- | -- | --
1 | -Start cpu stress on host. TICKScript is not modified | 2+1 (cpu-total) | 3 | `OK` |`CRIT: Series – cpu0/myhost CRIT: Series – cpu1/myhost CRIT: Series – cpu-total/myhost`
2 | Stop cpu stress on host. TickScript is not modified | 2+1 (cpu-total) | 3 | `OK` |`OK: Series  - cpu0/myhost OK: Series – cpu1/myhost OK: Series – cpu-total/myhost`
3 | Modify TICKScript, setup threshold to fire CRITS | 2+1 (cpu-total) | 3 | `OK` |`CRIT: Series – cpu0/myhost CRIT: Series – cpu1/myhost CRIT: Series – cpu-total/myhost`
4 | Modify TICKscript, setup threshold to fire OK after CRITS | 2+1 (cpu-total) | 1 | `NOOK` |`OK: Series – (RANDOM?)/myhost`

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the reload scenario with the supplied TICKScript on Kapacitor 1.5.0: create multiple alert series, change thresholds, and reload the task to force OK states. Trace alert-node state handling during reload and compare emitted events with the expected N OK events. Done means every active series emits its own OK event after the reload.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.