influxdata / influxdata/kapacitor
Monitor Kapacitor by another Kapacitor
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
We have 2 Kapacitor (one on our premise and another is managed by Influxdata in the Cloud)
We rely on our on-premise Kapacitor and we want the cloud kapacitor to notify us
I used a Deadman alert on `_kapacitor.autogen._stats` to check if data become not available for longer than 1 min
I set this alert on the cloud kapacitor and resulted in this Tickscript
```
var db = '_kapacitor'
var rp = 'autogen'
var measurement = 'stats_'
var groupBy = []
var whereFilter = lambda: TRUE
var period = 1m
var name = 'Kapacitor Down'
var idVar = name + ':{{.Group}}'
var message = 'I\'m down!'
var idTag = 'alertID'
var levelTag = 'level'
var messageField = 'message'
var durationField = 'duration'
var outputDB = 'chronograf'
var outputRP = 'autogen'
var outputMeasurement = 'alerts'
var triggerType = 'deadman'
var threshold = 0.0
var data = stream
|from()
.database(db)
.retentionPolicy(rp)
.measurement(measurement)
.groupBy(groupBy)
.where(whereFilter)
var trigger = data
|deadman(threshold, period)
.stateChangesOnly()
.message(message)
.id(idVar)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
.slack()
.channel('#kapacitor-test')
.iconEmoji(':cry:')
.username('Kapacitor Test')
trigger
|eval(lambda: "emitted")
.as('value')
.keep('value', messageField, durationField)
|eval(lambda: float("value"))
.as('value')
.keep()
|influxDBOut()
.create()
.database(outputDB)
.retentionPolicy(outputRP)
.measurement(outputMeasurement)
.tag('alertName', name)
.tag('triggerType', triggerType)
trigger
|httpOut('output')
```
But the alert is getting triggered although our Kapacitor is running and sending data into `_kapacitor` database
Contributor guide
Research direction
Start by reviewing the supplied Tickscript, especially its `_kapacitor.autogen._stats` input and Deadman configuration, and compare the cloud and on-premise Kapacitor data sources. Done means identifying why the alert fires while the on-premise Kapacitor is still sending data and documenting the required configuration change.
Written by the indexing model from the issue text.
Assessment
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100