influxdata / influxdata/kapacitor
Backfill stream data on script start
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
When a script is started, it sometimes needs to accumulate certain amount of data before it can take action (e.g. alert). For stream scripts this means that you have to wait for a while.
It would instead be useful to have a feature on the `from()` node where when it first starts up, before accepting any data from the `stream` node, it runs a query against InfluxDB for data which matches the various `database()`/`measurement()`/`retentionPolicy()`/`where()`/... properties, along with an additional property `backfillTime()` (or whatever name) to specify how far back it should query. Then once it has finished processing all this data, it starts reading from the `stream` node.
It might also be useful to add a field to the emitted point indicating whether the point came from the backfill, or the `stream` node. This would be so that the user could check the value and avoid taking action on backfill data. However in the idea of what this proposal is meant to solve, this shouldn't be necessary. Since the idea is to provide enough data for action to be taken, then the user should be able to set the backfill duration to where the script won't take action because it hasn't acquired enough data yet.
Contributor guide
Research direction
Start at the from() node and the stream-script processing path, then review how database(), measurement(), retentionPolicy(), and where() define the InfluxDB query. Done means the node can backfill the requested time range before consuming stream data, with any emitted backfill indicator defined consistently.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- databases, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100