Heartbeat (Webhook) stuck after opsgenie connection issue

What did you do?
Recently we started to get regular opsgenie heartbeat expired alerts.
The logs of alertmanager indicated that there was an interim issue connecting to opsgenie.
Maybe opsgenie is less reliable recently which revealed an bug that exists for a long time.

Alertmanager stopped sending heartbeats/alerts via the webhook integration at all after this issue. Restarting Alertmanager solved the issue.
We've also observed the same issue using webhook on another setup that is unrealted to opsgenie.

What did you expect to see?
Alertmanager webhook integration should recover from connection issues.
What did you see instead? Under which circumstances?
Alertmanager webhook integration did not recover from the issue itself and required a restart to recover.
Environment
kube-prometheus-stack, victoria metrics and alertmanager version 0.25.0 and 0.26.0 are affected

• System information:

  Kubernetes / GKE and Rancher / RKE2

• Alertmanager version:

  Initially we observed the issue with alertmanager 0.25.0 and upgraded to 0.26.0 hoping to solve the issue.
  But 0.26.0 showed the exact same error.
  version="(version=0.25.0, branch=HEAD, revision=258fab7cdd551f2cf251ed0348f0ad7289aee789)
  version="(version=0.26.0, branch=HEAD, revision=d7b4f0c7322e7151d6e3b1e31cbc15361e295d8d)"
  

• Prometheus version:

  Affected promtheus and victoria metrics setups.

• Logs:

ts=2024-01-05T20:52:00.263Z caller=notify.go:757 level=info component=dispatcher receiver=opsgenie.heartbeat integration=webhook[0] aggrGroup="{}/{alertname=~\"Watchdog|InfoInhibitor\"}:{alertname=\"Watchdog\", cluster=\"redacted by me\"}" msg="Notify success" attempts=2
ts=2024-01-05T20:51:59.463Z caller=notify.go:745 level=warn component=dispatcher receiver=opsgenie.heartbeat integration=webhook[0] aggrGroup="{}/{alertname=~\"Watchdog|InfoInhibitor\"}:{alertname=\"Watchdog\", cluster=\"redacted by me\"}" msg="Notify attempt failed, will retry later" attempts=1 err="Post \"<redacted>\": read tcp 172.16.0.20:44786->52.84.251.74:443: read: connection reset by peer"

Sometimes these logs even contain an http status page returned by opsgenie but that would be too noisy to post here.

No source files or tests are named. Start by tracing Alertmanager's webhook notification retry and recovery path around the logged connection-reset failure; done means webhook heartbeats and alerts resume after a transient connection issue without restarting Alertmanager, with a regression test covering recovery.