influxdata / influxdata/kapacitor

OpsGenie2 Integration: teams() not working, if team derived from tags

Open
#2,410 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

We are currently moving from Slack Notifications to use Kapacitor with the OpsGenie Integration. We have the following TICKscript template "generic_value_alert.tick":
```
dbrp "telegraf"."retain30days"

// Which measurement to consume
var measurement string

// Optional list of group by dimensions
var groups = [*]

// Optional where filter
var whereFilter = lambda: TRUE

// function to compute the metric from the measurement
var metric lambda

// slack channel to post the alert to
var slackChannel = '#alertfeed'

// The threshold we want to alert on
var critLevel = 90

// message to post on alert
var message string

// name of the alert
var name string

var opteam = 'core'

var idVar = name + ' {{.Group}}'
var idTag = 'alertID'
var levelTag = 'level'
var messageField = 'message'
var durationField = 'duration'
var outputDB = 'chronograf'
var outputRP = 'autogen'
var outputMeasurement = 'alerts'
var triggerType = 'threshold'

var data = stream
|from()
.measurement(measurement)
.groupBy(groups)
.where(whereFilter)
|eval(metric)
.as('value')

var trigger = data
|alert()
.crit(lambda: "value" > critLevel)
.message(message)
.id(idVar)
.idTag(idTag)
.levelTag(levelTag)
.messageField(messageField)
.durationField(durationField)
.stateChangesOnly()
.slack()
.channel(slackChannel)
.opsGenie2()
.teams(opteam)

trigger
|eval(lambda: float("value"))
.as('value')
.keep()
|influxDBOut()
.create()
.database(outputDB)
.retentionPolicy(outputRP)
.measurement(outputMeasurement)
.tag('alertName', name)
.tag('triggerType', triggerType)

trigger
|httpOut('output')
```

and the following task definition in "disk_space_low.json":
```
{
"template-id": "generic_value_alert",
"vars": {
"name": {
"type": "string",
"value": "Disk: space used > 85%"
},
"measurement": {
"type": "string",
"value": "disk"
},
"metric": {
"type": "lambda",
"value": "\"used_percent\""
},
"critLevel" : {
"type": "int",
"value": 85
},
"groups": {
"type": "list",
"value": [
{
"type": "string",
"value": "host"
},
{
"type": "string",
"value": "path"
},
{
"type": "string",
"value": "opteam"
}
]
},
"opteam": {
"type": "string",
"value": "{{ index .Tags \"opteam\" }}"
},
"message": {
"type": "string",
"value": "*{{.Level}} - {{.ID}}*\nEvent Time: {{.Time}}\nDisk Path: {{ index .Tags \"path\" }}\nDisk Usage: {{ index .Fields \"value\" | printf \"%0.2f\" }}%\nTeam: {{ index .Tags \"opteam\" }}"
}
}
}
```
Note: We are using Telgraf and InfluxDB where each VM that we monitor has a global_tag name "opteam" assigned to it. This tag defines which Team in OpsGenie will receive alerts from the VM.

**Expected behaviour**
When an alert is fired (Disk space on a VM is low) the alert should be sent to OpsGenie with the `opteam` correctly assigned.

**Current behaviour**
In the message that we send to Slack we see that "{{ index .Tags \"opteam\" }}" is correctly displayed. However, in OpsGenie the team is not assigned correctly.

**Kapacitor OSS 1.5.5 configuration**
```
[opsgenie2]
enabled = true
api-key = "XXXXX-XXXXXX-XXXX-XXX"
url = "https://api.opsgenie.com/v2/alerts"
recovery_action = "close"
global = false
```

Contributor guide

Open the contributing guide

Research direction

Start with the OpsGenie2 integration entry point used by `.opsGenie2().teams(opteam)` and reproduce the case from `generic_value_alert.tick`, `disk_space_low.json`, and the supplied `[opsgenie2]` configuration. Trace how the tag-derived `opteam` value is passed to the OpsGenie API, then verify that an alert assigns the expected team while Slack output remains correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.