influxdata / influxdata/kapacitor
Raise alert on same measurement but different tags
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
**Description**
I have a measurement which looks like this:
```
name: test
tags: status = (ok | nok)
field: number (int)
```
I need to raise an alert when the failure rate `nok / (ok+nok)` is > `Z%`
.
I first tried with `batch` using the following syntax:
```
var influx_query = 'SELECT count(number) AS count from "test"."autogen"."test"'
status=\'failed\''
var total_failed_query = 'SELECT count(number) AS failed_count from "test"."autogen"."test" WHERE status=\'failed\''
var total = batch
|query(influx_query)
.period(5m)
.every(1m)
.fill('none')
var failed = batch
|query(total_failed_query)
.period(period)
.every(every)
.fill(0)
failed
|join(total)
.as('failed','success')
.tolerance(5m)
.fill(0)
|eval(lambda: "failed.failed_count" / "total.count")
.as('failure_rate')
|alert()
.id('kapacitor_test')
.warn(...)
```
The thing is, when the query 2 is executed, it happens there are no point with `status` at `failed` . Due to this [influxdb issue](https://github.com/influxdata/influxdb/issues/6967), `fill(0)` returns no points causing the `join()` to never be evaluated.
As a result, if I was in warning mode and all new point have `status` at `success` I will never go back in normal level because the join() will not be triggered.
Do you have any recommendations to solve this kind of problem ? Is it possible to perform it in one query using groupBy ? (I tried also but failed as there is still no point for the `failed` tag).
Thanks.
Contributor guide
Research direction
No repository files or tests are named. Start by reproducing the batch queries, fill(0), groupBy, and join behavior described in the issue, then compare it with the linked InfluxDB issue. Done means establishing whether Kapacitor can evaluate the alert when a tag has no points and documenting or implementing a reliable recovery path.
Written by the indexing model from the issue text.
Assessment
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100