influxdata / influxdata/kapacitor

"Reduce" node and reacting to silent periods

Open
#1,304 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

We're wanting to do something like the following...

```
stream
| from()
.measurement('some_measurement')
| where(lambda: "some condition")
| stats(1s)
| derivative('emitted')
.unit(1s)
.nonNegative()
| window()
.periodCount('3600s')
.everyCount(1)
| reduce(lambda: \"emitted\" + \"_previous\") <-- new node type
.as("count")
| alert()
.crit(lambda: \"count\" > 0)
```

here "reduce" would take a batch, apply the lambda against each point, and supply intermediate values to the lambda as "_previous" (open to suggestion for a better name)

As for why we want this, we're trying to apply rules/tasks that react to messages/events/metrics that come and go (max-throughput, deadman, etc), we're having to deal with silent periods where the data we're trying to alert on doesn't just go down in frequency but goes also doesn't emit at all for extended periods.

We also have requirements to alert in a timely manner (e.g. evaluate max-throughput over a 1 hour window but alert within a few seconds of alert being triggered) that forces us to use a sliding window rather than stats based approach to evaluating alert conditions.

After some thought along the lines of joining the evaluated stream with a separate stats stream to force evaluation of the alert logic, we discarded this approach as it requires lambda expressions to be modified to account for the prefixes applied by the join node to each data field. Also creating a stream just to join it and discard it to force evaluation of the pipeline just felt... dirty.

We're willing to do the implementation of the node, we're looking for feedback as to whether this seems useful enough to include in kapacitor proper.

Contributor guide

Open the contributing guide

Research direction

No files or tests are named. Start by reviewing existing node types and the alert/window evaluation behavior, then clarify the proposed reduce semantics and handling of silent periods. Done means the project has decided whether this node belongs in Kapacitor and has an agreed implementation path.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.