influxdata / influxdata/kapacitor

Bug on outer join data (caching issue?)

Open
#1,754 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I am having irregular data points - error log with meric like this:
```
error_log,host=cache-base,severity=warning error="illegal block size"
```

As the errors data - irregular sequence, I should use somethimg like base signal to trigger alert back to OK states.

Documentation for [join node](https://docs.influxdata.com/kapacitor/v1.4/nodes/join_node/) says that:
> fill('null')
> null - fill missing points with null, full outer

So I have tick with join of base-signal (system mesurement) and irregular data - error_log measurement.

Here is simplified sample:
```go
dbrp "telegraf"."autogen"

var db = 'telegraf'
var rp = 'autogen'
var idVar = '{{ .TaskName }}:{{ index .Tags "host" }}'
var message = '{{ .Level }}: {{ index .Tags "host" }}. To many warnings count: {{ index .Fields "num_errors"}}.'
var period = 60m
var repeat = 1m
var critLevel = 100

var cache_servers = batch
|query('SELECT last("load1") as "load_level" FROM "telegraf"."autogen"."system" WHERE "host" =~ /cache/')
.period(period)
.every(repeat)
.groupBy('host')
.align()

var errors = batch
|query('SELECT count("error") as "num_errors" FROM "telegraf"."autogen"."error_log" WHERE "severity" = \'warning\' AND "host" =~ /cache/')
.period(period)
.every(repeat)
.groupBy('host')
.align()
.fill('none')

var data = cache_servers
|join(errors)
.as('servers','errors')
.tolerance(1m)
.fill('null')
|default()
.field('errors.num_errors', 0)
|eval(lambda: "errors.num_errors")
.as('num_errors')
|alert()
.crit(lambda: "num_errors" > critLevel)
.id(idVar)
.message(message)
.topic('warnings-state')
.log('/tmp/kapacitor_alerts.log')
```
Alert fails into critical state fine, but never goes back to OK state before kapacitor process stoped.
When no any data comes from errors batch, join would not be complete ever!
Looks like this issue is similar with #1704

Such behaviour of outer join absolutely discouraging!
Depending on documentation, probably, outer join should not wait for data from the right part of join beyond **tolerance(period)** when **fill('null')** property used.

I've crooked walkaround for this simplified case with reverse deadman switch based on **stats** with **derivative** , but it is almost impossible to use this walkaround for other cases when I am trying to observe some complex processes trough the couple of services.

Contributor guide

Open the contributing guide

Research direction

Start with the join node documentation and reproduce the supplied TICKscript using the cache_servers and errors batches, especially the tolerance and fill settings. Compare the behavior with issue #1704 and trace why the outer join does not complete when the errors stream is silent. Done means the alert can return to OK without stopping Kapacitor, with coverage for irregular right-side data.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.