intermittent bogus resolved notifications

What did you do?

Configured this Prometheus rule which triggers correctly and (through alertmanager) notifies us when the alarm is firing.

expr: package_long_jobs * on (SystemName,PackageName) group_left(package_runtime_running_time) package_runtime_running_time > 0

What did you expect to see?

No "resolved" notifications until the rule evaluates false.

What did you see instead? Under which circumstances?

Sometimes we get a bogus "resolved" notification, and shortly after that (within 5 minutes) the alarm is back in the "firing" state. Sometimes this happens multiple times over a longer period, like an alarm that fires for 20 hours, then resolves and immediately fires again, continues to fire for a few more hours, then resolves and immediately is in firing state again.
It sounds similar to #952

When I plot the rule expression in the prometheus UI I have verified that the expression ALWAYS evaluates to true
over the entire period. So I don't understand what the "intermittent" resolved notifications are coming.

To debug I started querying the /api/v1/alerts endpoint (every 5 mins) to look at the startsAt,endsAt times of the alarms
and noticed that when we get the bogus resolved notifications it seems that the /api/v1/alerts API stops listing the alarm in the json result. Then shortly after that, the response lists the same alarm again but with a new startsAt endsAt times.

Here is an example showing the results returned from the /api/v1/alerts API.
The csv fields below are:
| time I polled the api | fingerprint | startsAt time | endsAt time | alertname |

Why is difference between startsAt and endsAt in the "new alarm" only 8 minutes vs 2 hours in the "old alarm"

2021-04-21T13:35:01.657039+00:00Z,50bcd095389cc2d0,2021-04-21T11:39:50.684Z,2021-04-21T13:39:50.684Z,"Long Running Packages"
2021-04-21T13:35:01.657039+00:00Z,cfd192b0696b111e,2021-04-21T11:39:50.684Z,2021-04-21T13:39:50.684Z,"Long Running Packages"
2021-04-21T13:40:01.371449+00:00Z,50bcd095389cc2d0,2021-04-21T11:39:50.684Z,2021-04-21T13:47:50.684Z,"Long Running Packages"
2021-04-21T13:40:01.371449+00:00Z,cfd192b0696b111e,2021-04-21T11:39:50.684Z,2021-04-21T13:47:50.684Z,"Long Running Packages"
…<no mention of Long Running Packages alarm>….
2021-04-21T14:00:01.539915+00:00Z,50bcd095389cc2d0,2021-04-21T13:59:50.684Z,2021-04-21T14:07:50.684Z,"Long Running Packages"
2021-04-21T14:00:01.539915+00:00Z,cfd192b0696b111e,2021-04-21T13:59:50.684Z,2021-04-21T14:07:50.684Z,"Long Running Packages"

Environment

• System information:

  Linux 4.2.0-42-generic x86_64

• Alertmanager version:

  alertmanager, version 0.21.0 (branch: HEAD, revision: 4c6c03ebfe21009c546e4d1e9b92c371d67c021d)
  build user: root@dee35927357f
  build date: 20200617-08:54:02
  go version: go1.14.4

• Prometheus version:

  prometheus, version 2.26.0 (branch: HEAD, revision: 3cafc58827d1ebd1a67749f88be4218f0bab3d8d)
  build user: root@a67cafebe6d0
  build date: 20210331-11:56:23
  go version: go1.16.2
  platform: linux/amd64

• Alertmanager configuration file:

global:
  smtp_smarthost: 'smtp-1.imy.domain.com:25'
  smtp_from: 'no-reply@imy.domain.com'
  smtp_require_tls: false
  resolve_timeout: 5m

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 1h
  receiver: 'Alerts-Prod'

  routes:
  - match:
      job: 'oo-metrics'
    receiver: 'Alerts-Prod'
    continue: true
    repeat_interval: 24h

receivers:
- name: 'Alerts-Prod'
  email_configs:
  - to: 'my-teams-channel@amer.teams.ms'
    send_resolved: true

templates:
- '/etc/alertmanager/templates/default.tmpl'

inhibit_rules:
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'dev', 'instance']

• Prometheus configuration file:

global:
  scrape_interval: 60s 
  evaluation_interval: 2m 
  scrape_timeout: 15s

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      - my-prom-host:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  - 'rules/*.yml'
scrape_configs:
  - job_name: 'mssql-exporter'
    metrics_path: /metrics
    scrape_interval: 300s
    static_configs:
    - targets: ['my-prom-host:9200']

• Prometheus runtime args:

    --config.file=/etc/prometheus/prometheus.yml \
    --storage.tsdb.retention.time=180d \
    --web.enable-lifecycle \
    --web.enable-admin-api \
    --log.format=logfmt -\
    --log.level=debug \
    --storage.tsdb.path=/data/prometheus/data \
    --web.external-url=http://my-prom-host:9090/prometheus \
    --web.route-prefix=/prometheus --rules.alert.resend-delay=2m

Start with the /api/v1/alerts endpoint and the alert resend behavior, then compare the reported lifecycle with the supplied Prometheus rule, evaluation intervals, and Alertmanager configuration. The issue mentions no source files or tests; done would require reproducing or explaining the intermittent resolved notifications and covering the behavior with a regression test.