Alert reach Opsgenie with different deduplicating aliases even in the presence of group_by labels

What did you do?
Our development teams all have a route they can configure, matched by a label only they use. They all have a configured receiveer that relies on opsgenie_configs to integrate with a Prometheus integration they own in opsgenie.

In addition to this we have one more "wire-tap" opsgenie integration which gets absolutely all the alerts.

So X teams and X+1 opsgenie_configs blocks.

To support this setup we are grouping alerts with group_byand 3 labels present on all alerts: alertname, environment and namespace.

Alerts are flowing to opsgenie, hitting botht he wire-tapping route and the one specific for the team owning the system.

What did you expect to see?
We expected the alert sent to the wire-tapping route and the team's route to have the exact same opsgenie alias, so they get deduplicated in opsgenie.

That's also what we understood should work based on the official documentation at https://prometheus.io/docs/alerting/latest/configuration/#route

What did you see instead? Under which circumstances?
Alerts going throug wire-tapping and those going to teams have different aliases, leading to what we consider duplicate alerts in Opsgenie.

Environment

• System information:
  N/A

• Alertmanager version:
  From alert manager status page

Version Information
Branch:HEAD
BuildDate:20221222-14:51:36
BuildUser:root@abe866dd5717
GoVersion:go1.19.4
Revision:258fab7cdd551f2cf251ed0348f0ad7289aee789
Version:0.25.0

• Prometheus version:
  From Prometheus status page (web)

Version: 2.42.0
Revision: 225c61122d88b01d1f0eaaee0e05b6f3e0567ac0
branch: HEAD
BuildUser: root@c67d48967507
BuildDate: 20230201-07:53:32
GoVersion: go1.19.5

• Alertmanager configuration file:
  Complete file is big, extracting the relavant parts here, more can be provided if necessary.

global:
  resolve_timeout: 5m
  http_config:
    follow_redirects: true
    enable_http2: true
  smtp_hello: localhost
  smtp_require_tls: true
  opsgenie_api_url: https://api.opsgenie.com/
  opsgenie_api_key_file: /etc/alertmanager/secrets/opsgenie/api_key
route:
  receiver: "null"
  group_by:
  - alertname
  - namespace
  - environment
  continue: false
  routes:
  - receiver: global-config/global-config-prom-rules-prometheus-rules-signicat/null
    group_by:
    - alertname
    - namespace
    - environment
    matchers:
    - namespace="global-config"
    continue: true
    routes:
    - receiver: serviceX/Opsgenie
      matchers:
      - severity="critical"
      - environment="prod"
      continue: true
  - receiver: opsgenie
    matchers:
    - severity="critical"
    - signicat_cloud="gke-prod"
    continue: true
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
inhibit_rules:
- source_matchers:
  - severity="critical"
  target_matchers:
  - severity=~"warning|info"
  equal:
  - namespace
  - alertname
- source_matchers:
  - severity="warning"
  target_matchers:
  - severity="info"
  equal:
  - namespace
  - alertname
- source_matchers:
  - alertname="InfoInhibitor"
  target_matchers:
  - severity="info"
  equal:
  - namespace
receivers:
- name: opsgenie
  opsgenie_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    api_key_file: /etc/alertmanager/secrets/opsgenie/api_key
    api_url: https://api.opsgenie.com/
    message: '{{ template "opsgenie.default.message" . }}'
    description: '{{ template "opsgenie.default.description" . }}'
    source: '{{ template "opsgenie.default.source" . }}'
- name: serviceX/Opsgenie
  opsgenie_configs:
  - send_resolved: true
    http_config:
      follow_redirects: true
      enable_http2: true
    api_key: <secret>
    api_url: https://api.opsgenie.com/
    message: '{{ template "opsgenie.default.message" . }}'
    description: '{{ template "opsgenie.default.description" . }}'
    source: '{{ template "opsgenie.default.source" . }}'
templates:
- /etc/alertmanager/config/*.tmpl

• Prometheus configuration file:
  N/A, can be disclosed if relevant

• Logs:
  N/A

Start by tracing Alertmanager route grouping and the Opsgenie notification entry point, using the supplied configuration to reproduce two matching routes with different aliases. Identify how group_by labels and separate opsgenie_configs affect alias generation, then add or update coverage so both routes produce the same alias and deduplicate in Opsgenie.