prometheus / prometheus/alertmanager
Receiver called with already notified resolved alerts
Open
Nobody has claimed this yet.
stale
- Dominant language
- Go
- Stars
- 8.6k
- Forks
- 2.5k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 61
Description
What did you do?
We have seen a problem with resolved alerts that are sent too many times.
We use one instance of prometheus and one instance of the alertmanager (no cluster).
What did you expect to see?
We are expecting to receive a message from the alertmanager with the servers down (alerts in firing status) and the servers available again (alerts in resolved status).
What is particularly interesting is when all the servers are available again (alerts group in resolved status).
What did you see instead? Under which circumstances?
When a server or several servers are available again, we receive several calls with the different resolved alerts.
We should receive the alert resolved information only once.
We only have one alert received by message (even if two alerts are resolved, see the last action).
Environment
-
System information:
Linux 3.10.0-957.1.3.el7.x86_64 x86_64
-
Alertmanager version:
0.16.1
-
Prometheus version:
2.7.1
-
Alertmanager configuration file:
global:
resolve_timeout: 8737h
route:
group_by: ['alertname']
group_wait: 0s
group_interval: 60s
repeat_interval: 8737h
receiver: 'web.hook'
receivers:
- name: 'web.hook'
webhook_configs:
- url: 'http://localhost:9991/v1/alerts/'
send_resolved: true
Please, note that :
- we use a custom exporter that run a scenario to check state of each server
- we do not need resolve timeout an repeat interval, so we use a big value as suggested here : https://github.com/prometheus/alertmanager/issues/935)
- Prometheus configuration file:
global:
scrape_interval: 5s
evaluation_interval: 20s
scrape_configs:
- job_name: 5c5d8e41c680703f3a26fb68-monitoring-1
scrape_interval: 60s
scrape_timeout: 10s
metrics_path: /metrics
static_configs:
- targets:
- server01
- server02
- server03
- server04
- server05
relabel_configs:
- source_labels:
- __address__
target_label: __param_target
- source_labels:
- __param_target
target_label: instance
- target_label: __address__
replacement: 'http-scenario-exporter:9900'
rule_files:
- rules/5c5d8e41c680703f3a26fb68-monitoring-1.yml
- Rule file
groups:
- name: 5c5d8e41c680703f3a26fb68-monitoring-1
rules:
- alert: 5c5d8e41c680703f3a26fb68-monitoring-1-0
expr: >-
http_scenario_ok{job='5c5d8e41c680703f3a26fb68-monitoring-1'}
== 0
for: 60s
labels: {}
annotations: {}
- Logs:
Here is how we reproduce this issue (note that the instanceName tag is set by the exporter).
server01, server02, server03, server04, server05 are up.
I stop server05 :
{
"level": 10,
"time": "2019-02-11T13:51:25.895Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
I stop server04 :
{
"level": 10,
"time": "2019-02-11T13:57:45.994Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
I restart server5 :
{
"level": 10,
"time": "2019-02-11T14:01:25.998Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "2019-02-11T14:01:25.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
I stop server03 : (here I receive two calls (!) : one with server04+server03 firing and the other with server04+server03 firing and server05 resolved !)
{
"level": 10,
"time": "2019-02-11T14:03:46.001Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:04:06.002Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "2019-02-11T14:01:25.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
I restart server04 : (here, I receive a lot of calls (!). Always with server03 firing.
Sometimes with server04 resolved and sometimes with server05 resolved.)
{
"level": 10,
"time": "2019-02-11T14:07:46.002Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "2019-02-11T14:07:45.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:08:06.004Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "2019-02-11T14:01:25.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:09:06.005Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "2019-02-11T14:07:45.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:09:26.004Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "2019-02-11T14:01:25.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1",
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:10:26.005Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "2019-02-11T14:07:45.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:11:46.005Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "2019-02-11T14:07:45.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1",
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:12:06.005Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "2019-02-11T14:01:25.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:13:06.013Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations":
{
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "2019-02-11T14:07:45.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations":
{
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:13:26.007Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations":
{
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations":
{
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "2019-02-11T14:01:25.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:14:26.007Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations":
{
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "2019-02-11T14:07:45.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations":
{
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:14:46.007Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations":
{
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations":
{
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "2019-02-11T14:01:25.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:15:46.008Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations":
{
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "2019-02-11T14:07:45.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations":
{
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:16:06.012Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations":
{
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server05",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations":
{
},
"_startsAt": "2019-02-11T13:51:25.888Z",
"_endsAt": "2019-02-11T14:01:25.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
{
"level": 10,
"time": "2019-02-11T14:17:06.009Z",
"pid": 1,
"hostname": "8f80b4c90c57",
"module": "DispatcherService",
"method": "dispatchAlert",
"_receiver": "web\\.hook",
"_status": "firing",
"_externalURL": "http://5743407ed2a5:9093",
"_version": "4",
"_groupKey": "{}:{alertname=\"5c5d8e41c680703f3a26fb68-monitoring-1-0\"}",
"_alerts": [{
"_status": "resolved",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server04",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations":
{
},
"_startsAt": "2019-02-11T13:57:45.888Z",
"_endsAt": "2019-02-11T14:07:45.888Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
},
{
"_status": "firing",
"_labels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"instanceName": "server03",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_annotations": {
},
"_startsAt": "2019-02-11T14:03:45.888Z",
"_endsAt": "0001-01-01T00:00:00.000Z",
"_generatorURL": "http://ddf2804e6999:9090/graph?g0.expr=http_scenario_ok%7Bjob%3D%225c5d8e41c680703f3a26fb68-monitoring-1%22%7D+%3D%3D+0&g0.tab=1"
}],
"_groupLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0"
},
"_commonLabels": {
"alertname": "5c5d8e41c680703f3a26fb68-monitoring-1-0",
"datacenter": "central",
"job": "5c5d8e41c680703f3a26fb68-monitoring-1"
},
"_commonAnnotations": {
},
"v": 1
}
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the supplied single-instance Alertmanager scenario and inspect the DispatcherService dispatchAlert logs, focusing on grouping, resolved-alert handling, and webhook notification timing. Done means the configured webhook receives resolved alert information only once for the described sequence, with regression coverage for the reproduced behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100