Alert not send

What did you do?

We have an alert with 3 webhook receivers.

The alert was sent yesterday evening,
then again today.

But the alerts of today were not sent to the webhook receivers.

What did you expect to see?

Calls to webhooks yesterday and today.

What did you see instead? Under which circumstances?

The webhooks were only called yesterday.

Environment

• Alertmanager version:

  0.18.0

• Prometheus version:

  2.11 and 2.12

Deserialization of NFLOG during the second incident (where we did not receive notifications):

Entry: {}/{recipient=~"^(?:(.*,)?appteam/ticket(,.*)?)$"}/{repeat_interval=""}:{alertid="XXX-00014", customer_name="XXX", env="prod", hostname="XXX-mtprd01", priority="P1", recipient="appteam/ticket,XXX/circuit", title="JVM Down"}:appteam/ticket/webhook/0
{
  "entry": {
    "groupKey": "XXX",
    "receiver": {
      "groupName": "appteam/ticket",
      "integration": "webhook"
    },
    "timestamp": "2019-09-24T15:53:55.222089917Z",
    "firingAlerts": [
      "8684883655238988612"
    ]
  },
  "expiresAt": "2019-09-29T15:53:55.222089917Z"
}

Deserialization AFTER today event is resolved:

Entry: {}/{recipient=~"^(?:(.*,)?appteam/ticket(,.*)?)$"}/{repeat_interval=""}:{alertid="XXX-00014", customer_name="XXX", env="prod", hostname="XXX-mtprd01", priority="P1", recipient="appteam/ticket,XXX/circuit", title="JVM Down"}:appteam/ticket/webhook/0
{
  "entry": {
    "groupKey": "XXX",
    "receiver": {
      "groupName": "appteam/ticket",
      "integration": "webhook"
    },
    "timestamp": "2019-09-25T15:20:55.074281100Z",
    "resolvedAlerts": [
      "8684883655238988612"
    ]
  },
  "expiresAt": "2019-09-30T15:20:55.074281100Z"
}

The timestamp of the first one is the BEGINNING of the first event.
The timestamp of the second one is the END of the second event.

I would have expected the first one to be the END of the first event?

Start by tracing Alertmanager's webhook notification and deduplication behavior using the two NFLOG entries and their timestamps. Reproduce the two-day alert sequence, then verify that webhook receivers are called for both incidents and that the recorded timestamps match the intended event boundaries.