influxdata / influxdata/kapacitor

Randomized batch evaluation time offset via Splay property

Open
#712 4 comments 0 reactions 0 assignees View on GitHub
new-feature
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

We have a bunch of alerts that get evaluated every X, where X is a multiple of 1 minute (could be as large as 24 hours). Looks like kapacitor always evaluates the batch at the XX:XX:00.000 mark, causing many alerts to be evaluated simultaneously at the minute mark followed by a long period of nothing. "Rounder" time intervals suffer more, e.g. at midnight every single alert is evaluated.
This is not currently a problem but it in the interests of load balancing it would be great to have a way to randomly offset the initial start time of some alert that runs every X minutes by a value Y where Y := random(0-1.0) \* X, so that alerts would be evaluated at time Y, X+Y, 2X+Y, etc.
This way the read load against the database would be more smoothly distributed.

Is this currently possible?

Contributor guide

Open the contributing guide

Research direction

The issue names no files or tests; start by locating Kapacitor's batch evaluation scheduler and its interval handling. Determine whether a configurable randomized initial offset can be supported, then verify the resulting X+Y schedule and document or test the behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.