influxdata / influxdata/kapacitor
Procstat alerts on all nodes when one influx enterprise node goes down
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
I have a tick script that looks like this:
```
batch
|query('''
SELECT count(pid) as count
from "sdp_monitoring"."13m"."procstat"
where "pattern" = 'kafka'
''')
.groupBy('cluster', 'host')
.period(45s)
.every(10s)
|deadman(0.0, 10s)
.stateChangesOnly()
.exec('/kapacitor/tick-scripts/hpomi.py', 'Kafka Process Down Alert', 'Kafka Process Down', 'Kafka Process Down')
```
and telegraf input that has this:
```
[agent]
interval = "10s"
[global_tags]
cluster = "{{ kafka_cluster }}"
broker = "{{ ansible_hostname }}"
[[inputs.procstat]]
pattern = "kafka"
[[inputs.cpu]]
percpu = true
totalcpu = true
[[inputs.disk]]
[[inputs.diskio]]
[[inputs.processes]]
[[inputs.mem]]
[[inputs.system]]
[[outputs.influxdb]]
urls = [ "{{ influx_cluster }}" ]
database = "monitoring"
retention_policy = "13m"
precision = "s"
```
I'm also running a 7 node influx cluster (3 meta and 4 data) and have a replication policy on the `monitoring` db as 2. I've simulated that when I take on influx node down, Kapacitor will throw a process down alert and then right after throw a process up alert on state change. The processes are fine, looks like its being triggered by just taking one influxdb node down.
Contributor guide
Research direction
Start with the supplied TICK script and Telegraf configuration, then reproduce the behavior by taking one InfluxDB cluster node down. Trace why the deadman alert changes state when process metrics are still present; this is done when node failure no longer produces false process-down and process-up alerts.
Written by the indexing model from the issue text.
Assessment
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100