influxdata / influxdata/kapacitor

Procstat alerts on all nodes when one influx enterprise node goes down

Open
#1,532 28 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

I have a tick script that looks like this:

```
batch
|query('''
SELECT count(pid) as count
from "sdp_monitoring"."13m"."procstat"
where "pattern" = 'kafka'
''')
.groupBy('cluster', 'host')
.period(45s)
.every(10s)
|deadman(0.0, 10s)
.stateChangesOnly()
.exec('/kapacitor/tick-scripts/hpomi.py', 'Kafka Process Down Alert', 'Kafka Process Down', 'Kafka Process Down')
```

and telegraf input that has this:

```
[agent]
interval = "10s"

[global_tags]
cluster = "{{ kafka_cluster }}"
broker = "{{ ansible_hostname }}"

[[inputs.procstat]]
pattern = "kafka"

[[inputs.cpu]]
percpu = true
totalcpu = true

[[inputs.disk]]

[[inputs.diskio]]

[[inputs.processes]]

[[inputs.mem]]

[[inputs.system]]

[[outputs.influxdb]]
urls = [ "{{ influx_cluster }}" ]
database = "monitoring"
retention_policy = "13m"
precision = "s"
```

I'm also running a 7 node influx cluster (3 meta and 4 data) and have a replication policy on the `monitoring` db as 2. I've simulated that when I take on influx node down, Kapacitor will throw a process down alert and then right after throw a process up alert on state change. The processes are fine, looks like its being triggered by just taking one influxdb node down.

Contributor guide

Open the contributing guide

Research direction

Start with the supplied TICK script and Telegraf configuration, then reproduce the behavior by taking one InfluxDB cluster node down. Trace why the deadman alert changes state when process metrics are still present; this is done when node failure no longer produces false process-down and process-up alerts.

Written by the indexing model from the issue text.

Assessment

Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.