influxdata / influxdata/kapacitor

Topic handlers don't trigger after a topic is deleted and automatically re-created

Open
#1,939 0 comments 2 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I wanted to clear out some old test events from some alert topics, and the only way I'm aware of to do this is to delete the topic. After I did, I noticed that the alert handlers attached to the topic weren't triggering anymore. I had to delete and re-define the handlers to get them to start triggering again.

Steps to replicate using the Kapcitor 1.5 container from Docker Hub:

1) Define a topic handler. I'm using SMTP in this example, but I saw the same behaviour with OpsGenie:
```
id: alert-email-test
topic: exampletopic
kind: smtp
options:
to:
- 'user@example.com'
```
2) Confirm that the topic handler sends emails when events are generated.

3) Delete the topic:
```
> kapacitor delete topics exampletopic
```
4) Trigger another event, which will recreate the topic automatically. The handlers appear attached to the topic, but no email is sent. There's also no email sent when the alert returns to the OK level:
```
> kapacitor show-topic exampletopic

ID: exampletopic
Level: CRITICAL
Collected: 1
Handlers: [alert-email-test]
Events:
Event Level Message
example/file_exists CRITICAL example/file_exists is CRITICAL
```
5) Delete and redefine the handler:
```
> kapacitor delete topic-handlers exampletopic alert-email-test
> kapacitor define-topic-handler alert-handler-email.yaml
```
6) Trigger another event and confirm that email is now sent.

I wanted to check if this was the intended behaviour, since if you let the topic get re-created by an event the handlers show up as attached but don't trigger. It's possible to work around this by redefining the topic handlers immediately after deleting a topic, which re-creates the topic and appears to trigger the handlers as expected.

Let me know if there's any further information that'd be helpful.

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with the Kapacitor 1.5 Docker container using the topic-handler definition and delete/recreate commands shown above. Trace topic and handler behavior across deletion and automatic recreation; done means the existing SMTP or OpsGenie handler triggers for subsequent alert and recovery events without being redefined.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.