influxdata / influxdata/kapacitor
Is there a recipe for catching Kapacitor up after a crash?
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
I'm trying to come up with a solution that will allow me to replace a crashed Kapacitor instance with a new Kapacitor instance and catch it up to where the old instance was at. If it crashed while it was in an alert state, I want to get it back into an alert state, starting at the original time that the alert would have started.
I believe there's a way to do this with replay, but I'm not entirely sure how. Here's the flow/problem I see happening (I apologize if I have some of the nomenclature wrong):
1. Kapacitor instance `A` is running with a `task`
1. `task` enters an alert state
1. `A` crashes
1. Time passes
1. I start up Kapacitor instance `B`
1. `B` starts getting data from Influx
1. I post `task` to `B`
1. The data still is such that `task` enters an alert state immediately
1. I replay relevant data.
1. From the replay data, `task` detects the original alert condition and triggers another alert based on the original time
Am I thinking about this correctly? Is there a way to handle this scenario so that only the original alert state will be triggered? I thought about preventing Influx from talking to `B` until I could run the replay, but in the time that it takes to run the replay, `task` might miss something.
Please let me know if you would like any clarification. I would appreciate any advice you have to offer. Also, if this is just crazy outside the use case for Kapacitor, feel free to tell me so.
Contributor guide
Research direction
Start by reviewing Kapacitor's replay behavior and the existing four-comment thread, focusing on recovery after a crash and preserving alert timing. Done means establishing whether a supported recovery recipe exists and documenting the required sequence, including how to avoid duplicate or missed alerts.
Written by the indexing model from the issue text.
Assessment
- Domain
- observability-sre
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100