influxdata / influxdata/kapacitor

Monitor Kapacitor by another Kapacitor

Open
#1,955 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

We have 2 Kapacitor (one on our premise and another is managed by Influxdata in the Cloud)

We rely on our on-premise Kapacitor and we want the cloud kapacitor to notify us

I used a Deadman alert on `_kapacitor.autogen._stats` to check if data become not available for longer than 1 min

I set this alert on the cloud kapacitor and resulted in this Tickscript

```
var db = '_kapacitor'

var rp = 'autogen'

var measurement = 'stats_'

var groupBy = []

var whereFilter = lambda: TRUE

var period = 1m

var name = 'Kapacitor Down'

var idVar = name + ':{{.Group}}'

var message = 'I\'m down!'

var idTag = 'alertID'

var levelTag = 'level'

var messageField = 'message'

var durationField = 'duration'

var outputDB = 'chronograf'

var outputRP = 'autogen'

var outputMeasurement = 'alerts'

var triggerType = 'deadman'

var threshold = 0.0

var data = stream
|from()
.database(db)
.retentionPolicy(rp)
.measurement(measurement)
.groupBy(groupBy)
.where(whereFilter)

var trigger = data
|deadman(threshold, period)
.stateChangesOnly()
.message(message)
.id(idVar)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
.slack()
.channel('#kapacitor-test')
.iconEmoji(':cry:')
.username('Kapacitor Test')

trigger
|eval(lambda: "emitted")
.as('value')
.keep('value', messageField, durationField)
|eval(lambda: float("value"))
.as('value')
.keep()
|influxDBOut()
.create()
.database(outputDB)
.retentionPolicy(outputRP)
.measurement(outputMeasurement)
.tag('alertName', name)
.tag('triggerType', triggerType)

trigger
|httpOut('output')

```

But the alert is getting triggered although our Kapacitor is running and sending data into `_kapacitor` database

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the supplied Tickscript, especially its `_kapacitor.autogen._stats` input and Deadman configuration, and compare the cloud and on-premise Kapacitor data sources. Done means identifying why the alert fires while the on-premise Kapacitor is still sending data and documenting the required configuration change.

Written by the indexing model from the issue text.

Assessment

Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.