influxdata / influxdata/influxdb

Alerts sometimes don't register

Open
#24,237 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
31.7k
Forks
3.7k
Avg merge
13h 37m
Merged PRs (30d)
8

Description

I'm currently using an InfluxDB OSS installation in a Kubernetes cluster (version v2.7.1) with the official helm chart. I've also set up an Edge Data Replication bucket to bring data from my home to my cloud, which is working perfectly so far.

However, I'm now looking to create alerts for when a value hasn't been emitted for a certain amount of time (e.g. 2 hours). The values come from my home-assistant Z-Wave network with the help of zwave-JS-ui. Sometimes, due to radio interference, the devices drop packets and un-join the Z-Wave network, which requires manual resetting. To address this issue, I want to receive a notification in slack to stay up-to-date.

I have set up a deadman check alert to check every minute if a value is emitted for the last 5 minutes (later I will increase the time as said above, when is stable). The problem is that in for some time is working and then somehow it stops registering that a value is critical. I dont have any error in the logs, only in the alert history page that is ok (which isn't). Is there something that am I missing? Is this common behavior.

Contributor guide

Open the contributing guide

Research direction

Start with the deadman check configuration, alert history page, and logs described in the issue, using the Kubernetes deployment and official Helm chart context. Reproduce the transition from working to non-critical behavior and document whether the alert state and Slack notification remain correct when values stop arriving.

Written by the indexing model from the issue text.

Assessment

Tech stack
helm, kubernetes
Domain
databases, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.