Long-term silence is ignored in some rare and unknown circumstances

What did you do?
We have a regex silence with the following matchers that ends at 00:00:00, 9999-01-01 (UTC)

cluster="(testcluster1|testcluster2)"

We have two alertmanagers that form a raft cluster.

Alertmanager sometimes, very rarely, still sends alerts to the webhook.

What did you expect to see?
I expect not receiving any alerts that match the silence.

What did you see instead? Under which circumstances?
Sometimes, under circumstances we do not understand yet, we still receive alerts. The silence seems to be ignored for a couple of seconds, and then alertmanager will apply it again.

It seems to make it less likely to add additional silences without the regular expressions.

Environment

• System information:

  Linux 4.15.0-132-generic x86_64

• Alertmanager version:

  alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03ebfe21009c546e4d1e9b92c371d67c021d)
  build user: root@dee35927357f
  build date: 20200617-08:54:02
  go version: go1.14.4

• Alertmanager configuration file:

global:
  # The smarthost and SMTP sender used for mail notifications.
  smtp_smarthost: 'XXX:587'
  smtp_from: 'XXX'
  smtp_auth_username: 'XXX'
  smtp_auth_password: 'XXX'

templates:
  - '/etc/prometheus/alertmanager_templates/*.tmpl'

route:
  group_by: ['alertname', 'dc']
  group_wait: 1m
  group_interval: 5m
  repeat_interval: 1d
  # Don't send SMS or E-Mail by default
  receiver: blackhole

  # Page team via SMS and Email if severity is critical,
  # Send only an email when severity is high.
  # Otherwise do nothing.
  routes:
  # Send a sms every 20 minutes
  - match:
      severity: critical
    repeat_interval: 20m
    receiver: team-sms
    continue: true
  # Send test alert sms immediately
  - match:
      alertname: test_alert_consul
    group_wait: 0s # send test alert immediately
    receiver: team-sms
    continue: true
  # Send an email for critical alerts as well
  - match:
      severity: critical
    receiver: team-email
    continue: true
  # Send email for high alerts after 1 day wating period
  # They are supposed to be fixed at office hours any way.
  - match:
      severity: high
      group_wait: 1d
    receiver: team-email

receivers:
- name: 'blackhole' # Use if you don't want to be notified
- name: 'team-email'
  email_configs:
    - to: 'XXX'
- name: 'team-sms'
  webhook_configs:
    - url: http://localhost:8834 # Our SMS alerting service is listening here

• Logs:
  There are no relevant logs. I will increase the log level and post something here once I found something relevant.

Start by reproducing the intermittent silence behavior with the provided regex, long expiration, and two-node Alertmanager raft cluster configuration. Trace silence application and cluster synchronization, then add a regression test demonstrating that matching alerts remain silenced during synchronization gaps.