Validation of successful alert delivery to Opsgenie

Hello,

I would like to inquire about a feature request for alert validation in Opsgenie receiver.

What happened and why?

In recent days, our team has been dealing with a situation regarding the delivery of alerts from Alertmanager to Opsgenie, where the Opsgenie backend subsequently discarded these alerts. When rejecting such alerts, it responded with a successful HTTP 202 - Accepted status.

In more detail:

• Opsgenie contains a Prometheus type integration. The token for this integration can be found in the Alertmanager configuration.

• Opsgenie has a UI setting in the integration called "Create alert," which has a mandatory field set by default as Alert message: [Prometheus] {{message}}. However, one of our teams inadvertently changed it to Alert message: {{title}}, and we were sending an empty {{title}} field.

  

• Opsgenie accepted such an alert normally and responded to Alertmanager with an HTTP 202 status.

• Only in the "Settings > Logs" section did we find the message: "Cannot validate alert. Reason: Property [message] cannot be empty."

• The raw data of the alert with the error message looked like this:

{
  "_result": {
    "alertAction": "create",
    "integrationName": "infra_labrador",
    "integrationActionName": "Create Alert"
  },
  "error": "com.opsgenie.exception.DiscardableValidationException: Can not validate alert. Reason: Property [message] cannot be empty.",
  "_incomingData": {
    "_parsedData": {
      "note": "null",
      "-responders": [],
      "-actions": [],
      "-tags": [],
      "description": "our message",
      "alias": "60b2f859ff07aa5989a50c6d3998793d46b1ea9825963ca4f829483e414a5481",
      "source": "alertmanager",
      "message": "firing 4 - etcd cluster has high number of failed grpc requests",
      "priority": "P2",
      "user": "Alert API",
      "entity": ""
    },
    "integrationType": "Prometheus",
    "integrationName": "infra_labrador",
    "incomingDataId": "0c94951b-dc0f-455d-a661-989d4c75d8c5"
  }
}

Why are we addressing this here?

Because Opsgenie Support responded that they might consider a feature request in the future that would validate whether such an alert can pass through their backend and potentially return an HTTP 4xx error.

https://jira.atlassian.com/browse/OPSGENIE-1755

We want to use this channel to inform other Opsgenie users who may encounter this issue. Additionally, we would like to request an upvote for this problem, as we believe it should be resolved on Opsgenie's side.

In the scenario I provided, currently, alerts that Opsgenie is unable to evaluate are only discovered later, but Opsgenie still returns an HTTP 202 to Alertmanager, giving users the impression that everything is fine. The state should definitely indicate that everything is not fine.

How can Alertmanager help?

Opsgenie can provide information after 5 minutes if the alert was successfully created and delivered. Alertmanager could validate this situation. Opsgenie responds to the created alert:

{
    "data": {
        "success": true,
        "action": "Create",
        "processedAt": "2017-05-24T14:24:20.844Z",
        "integrationId": "c9cec2cb-e782-4ebb-bc1d-1b2fa703cf03",
        "isSuccess": true,
        "status": "Created alert",
        "alertId": "8743a1b2-11da-480e-8493-744660987bef",
        "alias": "8743a1b2-11da-480e-8493-744660987bef"
    },
    "took": 0.022,
    "requestId": "ec7e1d8e-1c75-442e-a271-731070a7fa4d"
}

And Alertmanager can verify it using the API: https://docs.opsgenie.com/docs/alert-api#get-request-status
If it detects an error or something other than an HTTP 2xx status, we can subsequently create alerting for that.

Conclusion

You might be thinking that such a ticket doesn't belong here at all, and I understand that. However, it's a way to inform other users about this issue that affects Alertmanager. Furthermore, there are ways to try to address it.

Thank you for your time, hopefully, it will help someone else, and I would kindly ask for an upvote on the Opsgenie ticket. It would be a great help for us.

No files, tests, or entry points are named. Start by locating the Opsgenie receiver and reviewing how delivery responses are handled, then consult the linked Opsgenie status API documentation. Done means defining and implementing a supported validation flow for delayed delivery failures, with tests covering successful and unsuccessful status results.