influxdata / influxdata/kapacitor

Deadman alert fails for some hosts

Open
#2,347 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

Hi,

I have a pool of servers and I am gathering metrics from then and putting into an InfluxDB.
Now I created a deadman alert on chronograf that sends me alerts by email when one of my servers fail

but I think something is wrong

I always receive alerts about one of my servers that’s working fine
On the other hand, I’m not receiving alerts about some others servers when they’re down

What I must to do ?

My config
```
var db = 'telegraf'

var rp = 'XXX'

var measurement = 'cpu'

var groupBy = ['host']

var whereFilter = lambda: TRUE

var period = 10m

0s

var name = 'Server Down Alert'

var idVar = name + '-{{.Group}}'

var message = 'SERVER FAIL {{ index .Tags "host" }}'

var idTag = 'alertID'

var levelTag = 'level'

var messageField = 'message'

var durationField = 'duration'

var outputDB = 'chronograf'

var outputRP = 'autogen'

var outputMeasurement = 'alerts'

var triggerType = 'deadman'

var threshold = 0.0

var data = stream
|from()
.database(db)
.retentionPolicy(rp)
.measurement(measurement)
.groupBy(groupBy)
.where(whereFilter)

var trigger = data
|deadman(threshold, period)
.message(message)
.id(idVar)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
.stateChangesOnly()
.email()
.to('XXX@XXX')

trigger
|eval(lambda: "emitted")
.as('value')
.keep('value', messageField, durationField)
|eval(lambda: float("value"))
.as('value')
.keep()
|influxDBOut()
.create()
.database(outputDB)
.retentionPolicy(outputRP)
.measurement(outputMeasurement)
.tag('alertName', name)
.tag('triggerType', triggerType)

trigger
|httpOut('output')
```

Contributor guide

Open the contributing guide

Research direction

No repository file or test is named. Begin with the supplied Kapacitor deadman configuration, especially the host grouping, period, and stream source, and reproduce the differing behavior for reporting and non-reporting hosts. Done means identifying the cause and documenting or fixing it with a regression test.

Written by the indexing model from the issue text.

Assessment

Domain
databases, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.