influxdata / influxdata/kapacitor
Failed to send event to Alerta (alert implemented with batch data TICK script)
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
Hello influx team,
I created the below TICK script (**batch data**). It is a simple use case to test alerting using batch data on a single metric (taken from the available metrics of MongoDB input plugin).
**Metric:** commands_per_sec
**Alert trigger criterion:** commands_per_sec > 10
```
dbrp "telegraf"."autogen"
var groupBy = ['hostname']
var whereFilter = lambda: TRUE
var name = 'MongoDB Commands per sec'
var idVar = name + '-{{.Group}}'
var message = 'BATCH - High commands per sec {{ index .Fields "value" }} for {{ index .Tags "hostname" }}'
var idTag = 'alertID'
var levelTag = 'level'
var messageField = 'message'
var durationField = 'duration'
var outputDB = 'chronograf'
var outputRP = 'autogen'
var outputMeasurement = 'alerts'
var triggerType = 'threshold'
var data = batch
|query('SELECT commands_per_sec as value FROM "telegraf"."autogen"."mongodb" WHERE time >= now() - 30s GROUP BY "hostname"')
.period(30s)
.every(10s)
var trigger = data
|alert()
.crit(lambda: "value" > 10)
.message(message)
.id(idVar)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
.alerta()
.resource('{{ index .Tags "hostname" }}')
.event('Something went wrong')
.environment('Production')
.value('{{ index .Fields "value" }}')
.origin('kapacitor')
.services()
trigger
|eval(lambda: float("value"))
.as('value')
.keep()
|influxDBOut()
.create()
.database(outputDB)
.retentionPolicy(outputRP)
.measurement(outputMeasurement)
.tag('alertName', name)
.tag('triggerType', triggerType)
trigger
|httpOut('output')
```
The above TICK script has been saved to a file named: **mgdb_batch_alert.tick**, placed under the directory is used to mount the user data to the "Kapacitor" docker container.
Inside the docker container, alert defined and enabled in "Kapacitor" successfully:
```
kapacitor define mgdb_batch_alert ./mgdb_batch_alert.tick
kapacitor enable mgdb_batch_alert
kapacitor watch mgdb_batch_alert
```
As it is seen from the below log (output from kapacitor watch mgdb_batch_alert) the alert triggers and it is raised (could also see it in Alert History in Chronograf) however it fails to deliver it to Alerta with the error message:
_"Resource and Event are required to send an alert"_
However both **Resource** and **Event** have been defined in the TIK script. This error message has been traced to [ source code](https://github.com/influxdata/kapacitor/blob/master/services/alerta/service.go), but it was not possible to determine the root cause.
```
ts=2020-06-18T09:58:53.853Z lvl=debug msg="alert triggered" service=kapacitor
task_master=main task=batch_alert node=influxdb_out4 level=CRITICAL
id="BATCH - MongoDB Commands per sec-nil"
event_message="BATCH - High commands per sec 13 for " data="&{mongodb map[]
[time value] [[2020-06-18 09:58:25 +0000 UTC 4] [2020-06-18 09:58:25 +0000 UTC 4]
[2020-06-18 09:58:35 +0000 UTC 13] [2020-06-18 09:58:35 +0000 UTC 11]]}"
ts=2020-06-18T09:58:53.854Z lvl=error msg="failed to send event to Alerta"
service=alerta task=batch_alert err="Resource and Event are required to send an alert"
```
I do see an obvious reason why the alert using batch data should fail to show up in Alerta.
Could you please suggest what the issue may be and why it failed to send the event to Alerta?
Contributor guide
Research direction
Reproduce the alert with the provided mgdb_batch_alert.tick script, then inspect services/alerta/service.go and the Alerta alert path. Trace how Resource and Event are populated for batch data, and consider the issue done when the reproduced alert is delivered successfully without the reported validation error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, mongodb
- Domain
- backend, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100