influxdata / influxdata/kapacitor
Randomized batch evaluation time offset via Splay property
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
We have a bunch of alerts that get evaluated every X, where X is a multiple of 1 minute (could be as large as 24 hours). Looks like kapacitor always evaluates the batch at the XX:XX:00.000 mark, causing many alerts to be evaluated simultaneously at the minute mark followed by a long period of nothing. "Rounder" time intervals suffer more, e.g. at midnight every single alert is evaluated.
This is not currently a problem but it in the interests of load balancing it would be great to have a way to randomly offset the initial start time of some alert that runs every X minutes by a value Y where Y := random(0-1.0) \* X, so that alerts would be evaluated at time Y, X+Y, 2X+Y, etc.
This way the read load against the database would be more smoothly distributed.
Is this currently possible?
Contributor guide
Research direction
The issue names no files or tests; start by locating Kapacitor's batch evaluation scheduler and its interval handling. Determine whether a configurable randomized initial offset can be supported, then verify the resulting X+Y schedule and document or test the behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100