influxdata / influxdata/kapacitor

Backfill stream data on script start

Open
#1,468 1 comment 1 reaction 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

When a script is started, it sometimes needs to accumulate certain amount of data before it can take action (e.g. alert). For stream scripts this means that you have to wait for a while.

It would instead be useful to have a feature on the `from()` node where when it first starts up, before accepting any data from the `stream` node, it runs a query against InfluxDB for data which matches the various `database()`/`measurement()`/`retentionPolicy()`/`where()`/... properties, along with an additional property `backfillTime()` (or whatever name) to specify how far back it should query. Then once it has finished processing all this data, it starts reading from the `stream` node.

It might also be useful to add a field to the emitted point indicating whether the point came from the backfill, or the `stream` node. This would be so that the user could check the value and avoid taking action on backfill data. However in the idea of what this proposal is meant to solve, this shouldn't be necessary. Since the idea is to provide enough data for action to be taken, then the user should be able to set the backfill duration to where the script won't take action because it hasn't acquired enough data yet.

Contributor guide

Open the contributing guide

Research direction

Start at the from() node and the stream-script processing path, then review how database(), measurement(), retentionPolicy(), and where() define the InfluxDB query. Done means the node can backfill the requested time range before consuming stream data, with any emitted backfill indicator defined consistently.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
databases, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.