influxdata / influxdata/kapacitor

Is there a recipe for catching Kapacitor up after a crash?

Open
#1,222 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I'm trying to come up with a solution that will allow me to replace a crashed Kapacitor instance with a new Kapacitor instance and catch it up to where the old instance was at. If it crashed while it was in an alert state, I want to get it back into an alert state, starting at the original time that the alert would have started.

I believe there's a way to do this with replay, but I'm not entirely sure how. Here's the flow/problem I see happening (I apologize if I have some of the nomenclature wrong):

1. Kapacitor instance `A` is running with a `task`
1. `task` enters an alert state
1. `A` crashes
1. Time passes
1. I start up Kapacitor instance `B`
1. `B` starts getting data from Influx
1. I post `task` to `B`
1. The data still is such that `task` enters an alert state immediately
1. I replay relevant data.
1. From the replay data, `task` detects the original alert condition and triggers another alert based on the original time

Am I thinking about this correctly? Is there a way to handle this scenario so that only the original alert state will be triggered? I thought about preventing Influx from talking to `B` until I could run the replay, but in the time that it takes to run the replay, `task` might miss something.

Please let me know if you would like any clarification. I would appreciate any advice you have to offer. Also, if this is just crazy outside the use case for Kapacitor, feel free to tell me so.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing Kapacitor's replay behavior and the existing four-comment thread, focusing on recovery after a crash and preserving alert timing. Done means establishing whether a supported recovery recipe exists and documenting the required sequence, including how to avoid duplicate or missed alerts.

Written by the indexing model from the issue text.

Assessment

Domain
observability-sre
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.