VictorOps Alerts Incorrectly Merged

What did you do?
Received an alert from an AlertManger instance which did not end up in VictorOps.

What did you expect to see?
Expected the alert to appear in VictorOps as a new incident, which would have been routed accordingly to the production on-call team

What did you see instead? Under which circumstances?
The alert has been merged in the same incident with another different unrelated alert, resulting in it being routed incorrectly (production on-call team was not notified of the alert)

The "other alert":

• Was triggered before the one in production and created its own VictorOps incident
• Came from a non-production environment, from a different cluster, from a different alert manager.

The only thing in common between them was the entity_id, which we suspect is the reason behind them ending up merged together. None of the other fields/properties were similar.

The production environments' configuration has not been modified in a very long time.
The development environment's configuration has recently been changed and had VictorOps enabled with its own routing key.

Environment

• System information:

Kubernetes 1.14 in both clusters.

• Alertmanager version:

0.19.0 in non-production
0.17.0 in production

In both cases installed via the Prometheus Operator Helm Chart.

• Alertmanager configuration file:
  Non-production config:

global:
  resolve_timeout: 5m
route:
  group_by: [namespace, alertname]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
  receiver: 'null'
  routes:
    - match:
        alertname: Watchdog
      receiver: 'null'
    - receiver: 'prometheus-msteams'
      continue: true
    - receiver: 'null'
      continue: false
      match:
        victorops: no
    - receiver: 'victor-ops'
      match:
        severity: critical
receivers:
- name: 'null'
- name: 'prometheus-msteams'
  webhook_configs:
    - send_resolved: true
      url: <redacted>
- name: victor-ops
  victorops_configs:
    - api_key: <redacted>
      routing_key: <development routing key>
      entity_display_name: '{{ .GroupLabels.alertname }} {{ .GroupLabels.namespace }}'
      message_type: '{{ .CommonLabels.severity }}'

Production config:

global:
  resolve_timeout: 5m
  http_config: {}
  smtp_hello: localhost
  smtp_require_tls: true
  pagerduty_url: https://events.pagerduty.com/v2/enqueue
  hipchat_api_url: https://api.hipchat.com/
  opsgenie_api_url: https://api.opsgenie.com/
  wechat_api_url: https://qyapi.weixin.qq.com/cgi-bin/
  victorops_api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
route:
  receiver: "null"
  group_by:
  - job
  routes:
  - receiver: "null"
    match:
      alertname: Watchdog
  - receiver: prometheus-msteams
    continue: true
  - receiver: victor-ops
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 12h
receivers:
- name: "null"
- name: prometheus-msteams
  webhook_configs:
  - send_resolved: true
    http_config: {}
    url: <redacted>
- name: victor-ops
  victorops_configs:
  - send_resolved: true
    http_config: {}
    api_key: <secret>
    api_url: https://alert.victorops.com/integrations/generic/20131114/alert/
    routing_key: <production routing key>
    message_type: CRITICAL
    state_message: '{{ template "victorops.default.state_message" . }}'
    entity_display_name: '{{ template "victorops.default.entity_display_name" . }}'
    monitoring_tool: '{{ template "victorops.default.monitoring_tool" . }}'
templates: []

Start by reproducing the reported behavior using the non-production and production Alertmanager configurations, focusing on the different group_by settings and shared entity_id. Compare how the VictorOps notifications are grouped and routed across Alertmanager 0.17.0 and 0.19.0. Done means unrelated alerts create separate VictorOps incidents and reach the intended production on-call route.