influxdata / influxdata/kapacitor
Deadman alert fails for some hosts
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Hi,
I have a pool of servers and I am gathering metrics from then and putting into an InfluxDB.
Now I created a deadman alert on chronograf that sends me alerts by email when one of my servers fail
but I think something is wrong
I always receive alerts about one of my servers that’s working fine
On the other hand, I’m not receiving alerts about some others servers when they’re down
What I must to do ?
My config
```
var db = 'telegraf'
var rp = 'XXX'
var measurement = 'cpu'
var groupBy = ['host']
var whereFilter = lambda: TRUE
var period = 10m
0s
var name = 'Server Down Alert'
var idVar = name + '-{{.Group}}'
var message = 'SERVER FAIL {{ index .Tags "host" }}'
var idTag = 'alertID'
var levelTag = 'level'
var messageField = 'message'
var durationField = 'duration'
var outputDB = 'chronograf'
var outputRP = 'autogen'
var outputMeasurement = 'alerts'
var triggerType = 'deadman'
var threshold = 0.0
var data = stream
|from()
.database(db)
.retentionPolicy(rp)
.measurement(measurement)
.groupBy(groupBy)
.where(whereFilter)
var trigger = data
|deadman(threshold, period)
.message(message)
.id(idVar)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
.stateChangesOnly()
.email()
.to('XXX@XXX')
trigger
|eval(lambda: "emitted")
.as('value')
.keep('value', messageField, durationField)
|eval(lambda: float("value"))
.as('value')
.keep()
|influxDBOut()
.create()
.database(outputDB)
.retentionPolicy(outputRP)
.measurement(outputMeasurement)
.tag('alertName', name)
.tag('triggerType', triggerType)
trigger
|httpOut('output')
```
Contributor guide
Research direction
No repository file or test is named. Begin with the supplied Kapacitor deadman configuration, especially the host grouping, period, and stream source, and reproduce the differing behavior for reporting and non-reporting hosts. Done means identifying the cause and documenting or fixing it with a regression test.
Written by the indexing model from the issue text.
Assessment
- Domain
- databases, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100