influxdata / influxdata/influxdb

Critical deadman alerts not sending notifications into slack

Open
#20,282 10 comments 0 reactions 0 assignees View on GitHub
area/ui monitoring-and-alerting team/ui wontfix
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

__Steps to reproduce:__
List the minimal actions needed to reproduce the behavior.

1. Create deadman alert for server in Influx v2.0.2 for linux metric
2. Set up slack notification endpoint
3. Create two notification rules for slack; When status changes from ANY to CRIT & when status changes from ANY to OK
4. Trigger deadman alert (systemctl stop telegraf)
5. Turn telegraf back on to clear deadman alert (reset to ok)

__Expected behavior:__
When telegraf switched off - Slack notification for agent offline (deadman critical)
When telegraf turned back on - Slack notification for agent online (deadman OK)

__Actual behavior:__
No dead man alert for CRIT sent to slack
dead man alert for OK was successfully sent to slack

__Environment info:__
Influx Version: InfluxDB 2.0.2 (git: 84496e507a) build_date: 2020-11-19T03:59:35Z

Contributor guide

Open the contributing guide

Research direction

Reproduce the deadman alert flow with InfluxDB 2.0.2, Telegraf, systemctl, and the Slack notification endpoint described in the steps. Trace the critical and OK notification paths; done means Slack receives notifications when the agent goes offline and when it returns online.

Written by the indexing model from the issue text.

Assessment

Tech stack
linux
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.