influxdata / influxdata/kapacitor
Bug on outer join data (caching issue?)
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
I am having irregular data points - error log with meric like this:
```
error_log,host=cache-base,severity=warning error="illegal block size"
```
As the errors data - irregular sequence, I should use somethimg like base signal to trigger alert back to OK states.
Documentation for [join node](https://docs.influxdata.com/kapacitor/v1.4/nodes/join_node/) says that:
> fill('null')
> null - fill missing points with null, full outer
So I have tick with join of base-signal (system mesurement) and irregular data - error_log measurement.
Here is simplified sample:
```go
dbrp "telegraf"."autogen"
var db = 'telegraf'
var rp = 'autogen'
var idVar = '{{ .TaskName }}:{{ index .Tags "host" }}'
var message = '{{ .Level }}: {{ index .Tags "host" }}. To many warnings count: {{ index .Fields "num_errors"}}.'
var period = 60m
var repeat = 1m
var critLevel = 100
var cache_servers = batch
|query('SELECT last("load1") as "load_level" FROM "telegraf"."autogen"."system" WHERE "host" =~ /cache/')
.period(period)
.every(repeat)
.groupBy('host')
.align()
var errors = batch
|query('SELECT count("error") as "num_errors" FROM "telegraf"."autogen"."error_log" WHERE "severity" = \'warning\' AND "host" =~ /cache/')
.period(period)
.every(repeat)
.groupBy('host')
.align()
.fill('none')
var data = cache_servers
|join(errors)
.as('servers','errors')
.tolerance(1m)
.fill('null')
|default()
.field('errors.num_errors', 0)
|eval(lambda: "errors.num_errors")
.as('num_errors')
|alert()
.crit(lambda: "num_errors" > critLevel)
.id(idVar)
.message(message)
.topic('warnings-state')
.log('/tmp/kapacitor_alerts.log')
```
Alert fails into critical state fine, but never goes back to OK state before kapacitor process stoped.
When no any data comes from errors batch, join would not be complete ever!
Looks like this issue is similar with #1704
Such behaviour of outer join absolutely discouraging!
Depending on documentation, probably, outer join should not wait for data from the right part of join beyond **tolerance(period)** when **fill('null')** property used.
I've crooked walkaround for this simplified case with reverse deadman switch based on **stats** with **derivative** , but it is almost impossible to use this walkaround for other cases when I am trying to observe some complex processes trough the couple of services.
Contributor guide
Research direction
Start with the join node documentation and reproduce the supplied TICKscript using the cache_servers and errors batches, especially the tolerance and fill settings. Compare the behavior with issue #1704 and trace why the outer join does not complete when the errors stream is silent. Done means the alert can return to OK without stopping Kapacitor, with coverage for irregular right-side data.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100