influxdata / influxdata/kapacitor
Topic handlers don't trigger after a topic is deleted and automatically re-created
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
I wanted to clear out some old test events from some alert topics, and the only way I'm aware of to do this is to delete the topic. After I did, I noticed that the alert handlers attached to the topic weren't triggering anymore. I had to delete and re-define the handlers to get them to start triggering again.
Steps to replicate using the Kapcitor 1.5 container from Docker Hub:
1) Define a topic handler. I'm using SMTP in this example, but I saw the same behaviour with OpsGenie:
```
id: alert-email-test
topic: exampletopic
kind: smtp
options:
to:
- 'user@example.com'
```
2) Confirm that the topic handler sends emails when events are generated.
3) Delete the topic:
```
> kapacitor delete topics exampletopic
```
4) Trigger another event, which will recreate the topic automatically. The handlers appear attached to the topic, but no email is sent. There's also no email sent when the alert returns to the OK level:
```
> kapacitor show-topic exampletopic
ID: exampletopic
Level: CRITICAL
Collected: 1
Handlers: [alert-email-test]
Events:
Event Level Message
example/file_exists CRITICAL example/file_exists is CRITICAL
```
5) Delete and redefine the handler:
```
> kapacitor delete topic-handlers exampletopic alert-email-test
> kapacitor define-topic-handler alert-handler-email.yaml
```
6) Trigger another event and confirm that email is now sent.
I wanted to check if this was the intended behaviour, since if you let the topic get re-created by an event the handlers show up as attached but don't trigger. It's possible to work around this by redefining the topic handlers immediately after deleting a topic, which re-creates the topic and appears to trigger the handlers as expected.
Let me know if there's any further information that'd be helpful.
Contributor guide
Research direction
Reproduce the issue with the Kapacitor 1.5 Docker container using the topic-handler definition and delete/recreate commands shown above. Trace topic and handler behavior across deletion and automatic recreation; done means the existing SMTP or OpsGenie handler triggers for subsequent alert and recovery events without being redefined.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100