influxdata / influxdata/kapacitor

Kapacitor Alerts HTTP API - alerts "min-level" filter behavioural issues

Open
#1,382 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
2.4k
Forks
479
Avg merge
4d 16h
Merged PRs (30d)
4

Description

HI,

Are any duration filters automatically applied on the API responses? We use the HTTP API to fetch all alert topics, and the events (https://docs.influxdata.com/kapacitor/v1.2/api/api/#alerts) with min-level=WARNING.

Eg:
1. `http://:9092/kapacitor/v1preview/alerts/topics?min-level=WARNING`
2. `http://:9092/kapacitor/v1preview/alerts/topics/hostalive_check/events?min-level=WARNING`

Recently, it has come to our notice that one CRITICAL alert was not included in the API call's response
i.e http://:9092/kapacitor/v1preview/alerts/topics/hostalive_check returns level as "INFO" as shown.
```json
{
"link": {
"rel": "self",
"href": "/kapacitor/v1preview/alerts/topics/hostalive_check"
},
"id": "hostalive_check",
"level": "INFO",
"collected": 6,
"events-link": {
"rel": "events",
"href": "/kapacitor/v1preview/alerts/topics/hostalive_check/events"
},
"handlers-link": {
"rel": "handlers",
"href": "/kapacitor/v1preview/alerts/topics/hostalive_check/handlers"
}
}
```
but `http://:9092/kapacitor/v1preview/alerts/topics/hostalive_check/events` actually returns all the alerts, out of which one is CRITICAL as shown below:
```json
{
"link": {
"rel": "self",
"href": "/kapacitor/v1preview/alerts/topics/hostalive_check/events"
},
"topic": "hostalive_check",
"events": [
{
"link": {
"rel": "self",
"href": "/kapacitor/v1preview/alerts/topics/hostalive_check/events/hostalive_check$..."
},
"id": "hostalive_check$...",
"state": {
"message": "...",
"details": "...",
"time": "2017-05-02T10:25:56Z",
"duration": "2562047h47m16.854775807s",
"level": "CRITICAL"
}
},
{
....
}
]
}
```

One thing to highlight is that the state of the alert has not changed for quite a few days & duration field shows a large value in hours. Is there any reason why it would get ignored as part of the "Topic Status" API? Also, why doesn't it show up when filtered based on min-level?

Since the alert state is correct , i have not added the TICK Scripts here. Let me know if it is needed.

Thanks,
Rashmi

Contributor guide

Open the contributing guide

Research direction

Start with the alerts API endpoints shown in the report: the topic status response and the topic events response, both with and without min-level=WARNING. Reproduce the hostalive_check case using the returned CRITICAL event and its long duration, then trace why topic status and filtering disagree. Done means the behavior is explained and the API responses consistently handle the alert level.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
api, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.