influxdata / influxdata/kapacitor

Raise alert on same measurement but different tags

Open
#1,194 0 comments 0 reactions 0 assignees View on GitHub
pm/consumability
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

**Description**
I have a measurement which looks like this:

```
name: test
tags: status = (ok | nok)
field: number (int)
```

I need to raise an alert when the failure rate `nok / (ok+nok)` is > `Z%`
.

I first tried with `batch` using the following syntax:

```
var influx_query = 'SELECT count(number) AS count from "test"."autogen"."test"'
status=\'failed\''
var total_failed_query = 'SELECT count(number) AS failed_count from "test"."autogen"."test" WHERE status=\'failed\''

var total = batch
|query(influx_query)
.period(5m)
.every(1m)
.fill('none')

var failed = batch
|query(total_failed_query)
.period(period)
.every(every)
.fill(0)

failed
|join(total)
.as('failed','success')
.tolerance(5m)
.fill(0)
|eval(lambda: "failed.failed_count" / "total.count")
.as('failure_rate')
|alert()
.id('kapacitor_test')
.warn(...)
```

The thing is, when the query 2 is executed, it happens there are no point with `status` at `failed` . Due to this [influxdb issue](https://github.com/influxdata/influxdb/issues/6967), `fill(0)` returns no points causing the `join()` to never be evaluated.

As a result, if I was in warning mode and all new point have `status` at `success` I will never go back in normal level because the join() will not be triggered.

Do you have any recommendations to solve this kind of problem ? Is it possible to perform it in one query using groupBy ? (I tried also but failed as there is still no point for the `failed` tag).

Thanks.

Contributor guide

Open the contributing guide

Research direction

No repository files or tests are named. Start by reproducing the batch queries, fill(0), groupBy, and join behavior described in the issue, then compare it with the linked InfluxDB issue. Done means establishing whether Kapacitor can evaluate the alert when a tag has no points and documenting or implementing a reliable recovery path.

Written by the indexing model from the issue text.

Assessment

Domain
observability-sre
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.