influxdata / influxdata/influxdb
Critical deadman alerts not sending notifications into slack
- Dominant language
- Rust
- Stars
- 31.7k
- Forks
- 3.7k
- Avg merge
- 13h 37m
- Merged PRs (30d)
- 8
Description
__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.
1. Create deadman alert for server in Influx v2.0.2 for linux metric
2. Set up slack notification endpoint
3. Create two notification rules for slack; When status changes from ANY to CRIT & when status changes from ANY to OK
4. Trigger deadman alert (systemctl stop telegraf)
5. Turn telegraf back on to clear deadman alert (reset to ok)
__Expected behavior:__
When telegraf switched off - Slack notification for agent offline (deadman critical)
When telegraf turned back on - Slack notification for agent online (deadman OK)
__Actual behavior:__
No dead man alert for CRIT sent to slack
dead man alert for OK was successfully sent to slack
__Environment info:__
Influx Version: InfluxDB 2.0.2 (git: 84496e507a) build_date: 2020-11-19T03:59:35Z
Contributor guide
Research direction
Reproduce the deadman alert flow with InfluxDB 2.0.2, Telegraf, systemctl, and the Slack notification endpoint described in the steps. Trace the critical and OK notification paths; done means Slack receives notifications when the agent goes offline and when it returns online.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100