influxdata / influxdata/kapacitor
Kapacitor Alerts HTTP API - alerts "min-level" filter behavioural issues
- Dominant language
- Go
- Stars
- 2.4k
- Forks
- 479
- Avg merge
- 4d 16h
- Merged PRs (30d)
- 4
Description
HI,
Are any duration filters automatically applied on the API responses? We use the HTTP API to fetch all alert topics, and the events (https://docs.influxdata.com/kapacitor/v1.2/api/api/#alerts) with min-level=WARNING.
Eg:
1. `http://:9092/kapacitor/v1preview/alerts/topics?min-level=WARNING`
2. `http://:9092/kapacitor/v1preview/alerts/topics/hostalive_check/events?min-level=WARNING`
Recently, it has come to our notice that one CRITICAL alert was not included in the API call's response
i.e http://:9092/kapacitor/v1preview/alerts/topics/hostalive_check returns level as "INFO" as shown.
```json
{
"link": {
"rel": "self",
"href": "/kapacitor/v1preview/alerts/topics/hostalive_check"
},
"id": "hostalive_check",
"level": "INFO",
"collected": 6,
"events-link": {
"rel": "events",
"href": "/kapacitor/v1preview/alerts/topics/hostalive_check/events"
},
"handlers-link": {
"rel": "handlers",
"href": "/kapacitor/v1preview/alerts/topics/hostalive_check/handlers"
}
}
```
but `http://:9092/kapacitor/v1preview/alerts/topics/hostalive_check/events` actually returns all the alerts, out of which one is CRITICAL as shown below:
```json
{
"link": {
"rel": "self",
"href": "/kapacitor/v1preview/alerts/topics/hostalive_check/events"
},
"topic": "hostalive_check",
"events": [
{
"link": {
"rel": "self",
"href": "/kapacitor/v1preview/alerts/topics/hostalive_check/events/hostalive_check$..."
},
"id": "hostalive_check$...",
"state": {
"message": "...",
"details": "...",
"time": "2017-05-02T10:25:56Z",
"duration": "2562047h47m16.854775807s",
"level": "CRITICAL"
}
},
{
....
}
]
}
```
One thing to highlight is that the state of the alert has not changed for quite a few days & duration field shows a large value in hours. Is there any reason why it would get ignored as part of the "Topic Status" API? Also, why doesn't it show up when filtered based on min-level?
Since the alert state is correct , i have not added the TICK Scripts here. Let me know if it is needed.
Thanks,
Rashmi
Contributor guide
Research direction
Start with the alerts API endpoints shown in the report: the topic status response and the topic events response, both with and without min-level=WARNING. Reproduce the hostalive_check case using the returned CRITICAL event and its long duration, then trace why topic status and filtering disagree. Done means the behavior is explained and the API responses consistently handle the alert level.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- api, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100