Proposal: Paginate through listAlertGroups API

ListAlertGroups API

Existing sorting behavior is: compare the label set of the alert group first:

If m has fewer labels than o, it is before o. If it has more, it is not.
If the number of labels is the same, the superset of all label names is sorted alphanumerically. The first differing label pair found in that order determines the outcome: If the label does not exist at all in m, then m is before o, and vice versa. Otherwise the label value is compared alphanumerically.
If m and o are equal, the method returns false.

If the alert group labels are the same, compare the receiver alphanumerically.

Paginate option:

Paginate through alert groups and only return first x alerts from every alert group

We will paginate on this ListAlertGroups API, we will only return first X (1000 maybe) alerts per alert group. If customer want to see more alerts inside of each alert group, they can call the listAlerts API with the filter like
/alertmanager/api/v2/alerts?filter=group%3Dhello1&receiver=example

API spec

• URL query parameters:
  • maxAlertGroups Integer. The maximum number of alert group to return in one ListAlertGroups operation. The range is 1–1000. Optional
  • maxAlertItem Integer. The maximum number of alerts returned under each alert group. default to 100 Optional
  • nextToken String. The token for the next set of items to return. (You received this token from a previous call.). Optional
  • active Boolean. If true, the returned list includes active alerts. The default is true. Optional
  • silenced Boolean. If true, the returned list includes silenced alerts. The default is true. Optional
  • inhibited Boolean. If true, the returned list includes inhibited alerts. The default is true. Optional
  • unprocessed Boolean. If true, the returned list includes unprocessed alerts. The default is true. Optional
  • filter An array of strings. A list of matchers to filter alerts by. Optional
  • receiver String. A regular expression matching receivers to filter alerts by. Optional
• Sample response

•     groups: [
          {
            "alerts": [
              {
                "annotations": {
                  "summary": ""
                },
                "endsAt": "2030-06-15T21:17:27.369Z",
                "fingerprint": "08f58215e96e440b",
                "receivers": [
                  {
                    "name": "example-sns"
                  }
                ],
                "startsAt": "2030-06-15T21:17:27.369Z",
                "status": {
                  "inhibitedBy": [],
                  "silencedBy": [],
                  "state": "active"
                },
                "updatedAt": "2023-03-15T09:35:46.195Z",
                "generatorURL": "http://localhost:8080/uri",
                "labels": {
                  "alertname": "0122de25-ac2a-49d2-89d4-ddf4b349ed9a",
                  "group": "hello1"
                }
              }
            ],
            "labels": {
              "group": "hello1"
            },
            "receiver": {
              "name": "example-sns"
            },
            "route": "{ROUTE_UUID}"
          },
          {
            "alerts": [],
            "labels": {
              "group": "hello2"
            },
            "receiver": {
              "name": "example-sns"
            },
            "route": "{ROUTE_UUID}"
          }
        ],
        nextToken:"123"
  

Pros

• Easy to implement
• We will add the pagination support for an existing API

Cons

• Need to change the API return response behavior(extra field for parameters)
  • We need to introduce an optional field to let customer choose if return truncated result
  • Or We need to do v3 of the API for backward compatibility and not breaking customer
• Need to change the existing sorting behavior of current listAlertGroups API

Do not paginate, only return first x alerts from every alert group

We will not paginate on this ListAlertGroups API, we will only return first X (1000 maybe) alerts per alert group. If customer want to see more alerts inside of each alert group, they can call the listAlerts API with the filter like
/alertmanager/api/v2/alerts?filter=group%3Dhello1&receiver=example

This will filter label group=hello1 and reciever in example

Pros

• Easy to implement
• Do not need to change the existing sorting behavior of current listAlertGroups API

Cons

• Need to change the API return response behavior
  • We need to introduce an optional field to let customer choose if return truncated result
  • Or We need to do v3 of the API for backward compatibility and not breaking customer

Brand new API to return the alert group without alerts

We will create a new API that will only return a list of alert group without the alerts, for example:

[
    {
        "labels": {
            "group": "hello1"
        },
        "receiver": {
            "name": "example-sns"
        }
    },
    {
        "labels": {
            "group": "hello2"
        },
        "receiver": {
            "name": "example-sns"
        }
    }
]

If customer want to see the alerts inside of each alert group, they can call the listAlerts API with the filter like
alertmanager/api/v2/alerts?filter=group%3Dhello1&receiver=example

This will filter label group=hello1 and reciever in example.

Pros

• No change for the existing listAlertGroups API

Cons

• Grafana need to do 2 level API call in order to render existing alert group page

Flatten the construct

We will have to add two more field in the alerts attribute to identify the alert group each alert is belong to, the construct from ListAlertGroups API result will turn into :

[
    {
    "annotations": {
        "summary": ""
    },
    "endsAt": "2030-06-15T21:17:27.369Z",
    "fingerprint": "08f58215e96e440b",
    "receivers": [
        {
        "name": "example-sns"
        }
    ],
    "startsAt": "2030-06-15T21:17:27.369Z",
    "status": {
        "inhibitedBy": [],
        "silencedBy": [],
        "state": "active"
    },
    "updatedAt": "2023-03-15T09:35:46.195Z",
    "generatorURL": "http://localhost:8080/uri",
    "labels": {
        "alertname": "0122de25-ac2a-49d2-89d4-ddf4b349ed9a",
        "group": "hello1"
    },
    "groupLabels": {
        "group": "hello1"
    },
    "groupReceiver": {
        "name": "example-sns"
    }
    }
]

Sorting will be sort by the alert group fingerprint first and then the alerts inside of alert group

PaginationToken

The pagination token will be the fingerprint of the start alert group + “_” + fingerprint of the start alert

Pros

• Return the whole information in single API

Cons

• Need to change the API return response structure
  • We need to do v3 of the API for backward compatibility and not breaking customer
• Need to change the existing sorting behavior of current listAlertGroups API

UI change

• Filter change Similar change that grafana need to do, switch to service side filtering
• Custom grouping change
• Silence, inhibited filtering change to switch to service side filtering

Start at the existing ListAlertGroups and listAlerts API entry points, using the proposed query parameters, response examples, and sorting rules as the requirements. Before implementation, choose among the four alternatives, settle backward-compatibility and pagination-token behavior, and define acceptance checks for pagination, filtering, sorting, and the UI changes.