prometheus / prometheus/alertmanager

one alert firing,but over two resloved message received

Open
#2,151 9 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/high availability component/notify kind/more-info-needed
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

What did you do?
i am using Alertmanager HA in my cluster, sometimes,when a large number of warnings in one group are triggered, i usually receive two resloved message but only one alert. of course，the second resloved message often appears in the second group interval which also has similar alert message.

What did you expect to see?
one alert has only one resloved message, if one group has alert message again, shouldn't have resloved message which has been sent.

What did you see instead? Under which circumstances?
in one group ,when have alert message again , the resloved message which has been sent also send again
Environment
two Alertmanager HA

  • Alertmanager version:

    Alertmanager: v0.20.0

  • Prometheus version:

    2.14

  • Alertmanager configuration file:

route:
      group_by: ['alertname']
      group_wait: 60s
      group_interval: 5m
      repeat_interval: 3h

args:
        - "--config.file=/etc/alertmanager/alertmanager.yml"
        - "--storage.path=/data"
        - "--web.listen-address=:9095"
        - "--cluster.listen-address=0.0.0.0:8003"
        - "--cluster.peer=172.21.164.20:8003"
        - "--cluster.peer-timeout=30s"
        - "--cluster.gossip-interval=50ms"
        - "--cluster.pushpull-interval=2s"
        - "--log.level=debug"
  • Prometheus configuration file:
scrape_interval: 30s
  scrape_timeout: 10s
  evaluation_interval: 15s
  • Logs:
level=debug ts=2019-12-31T06:10:54.577Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:10:54 [DEBUG] memberlist: Stream connection from=172.21.164.20:24116\n"
level=debug ts=2019-12-31T06:10:54.577Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:10:54 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.164.20:8003\n"
level=debug ts=2019-12-31T06:10:56.578Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:10:56 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.164.20:8003\n"
level=debug ts=2019-12-31T06:10:56.578Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:10:56 [DEBUG] memberlist: Stream connection from=172.21.164.20:24126\n"
level=debug ts=2019-12-31T06:10:58.579Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:10:58 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.164.20:8003\n"
level=debug ts=2019-12-31T06:10:58.579Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:10:58 [DEBUG] memberlist: Stream connection from=172.21.164.20:24146\n"
level=debug ts=2019-12-31T06:11:00.580Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:00 [DEBUG] memberlist: Stream connection from=172.21.164.20:24154\n"
level=debug ts=2019-12-31T06:11:00.580Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:00 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.164.20:8003\n"
level=debug ts=2019-12-31T06:11:02.582Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:02 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.164.20:8003\n"
level=debug ts=2019-12-31T06:11:02.582Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:02 [DEBUG] memberlist: Stream connection from=172.21.164.20:24166\n"
level=debug ts=2019-12-31T06:11:04.583Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:04 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.164.20:8003\n"
level=debug ts=2019-12-31T06:11:04.584Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:04 [DEBUG] memberlist: Stream connection from=172.21.164.20:24178\n"
level=debug ts=2019-12-31T06:11:06.585Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:06 [DEBUG] memberlist: Stream connection from=172.21.164.20:24188\n"
level=debug ts=2019-12-31T06:11:06.585Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:06 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.164.20:8003\n"
level=debug ts=2019-12-31T06:11:08.586Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:08 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.164.20:8003\n"
level=debug ts=2019-12-31T06:11:08.586Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:08 [DEBUG] memberlist: Stream connection from=172.21.164.20:24204\n"
level=debug ts=2019-12-31T06:11:08.613Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=qos-deployment-erro[36baa7e][active]
level=debug ts=2019-12-31T06:11:10.587Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:10 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.164.20:8003\n"
level=debug ts=2019-12-31T06:11:10.588Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:10 [DEBUG] memberlist: Stream connection from=172.21.164.20:24212\n"
level=debug ts=2019-12-31T06:11:12.207Z caller=dispatch.go:135 component=dispatcher msg="Received alert" alert=node-notready[10a8ef6][active]
level=debug ts=2019-12-31T06:11:12.588Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:12 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.164.20:8003\n"
level=debug ts=2019-12-31T06:11:12.589Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:12 [DEBUG] memberlist: Stream connection from=172.21.164.20:24224\n"
level=debug ts=2019-12-31T06:11:14.589Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:14 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.164.20:8003\n"
level=debug ts=2019-12-31T06:11:14.590Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:14 [DEBUG] memberlist: Stream connection from=172.21.164.20:24248\n"
level=debug ts=2019-12-31T06:11:16.591Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:16 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.164.20:8003\n"
level=debug ts=2019-12-31T06:11:16.591Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:16 [DEBUG] memberlist: Stream connection from=172.21.164.20:24252\n"
level=debug ts=2019-12-31T06:11:18.592Z caller=cluster.go:306 component=cluster memberlist="2019/12/31 06:11:18 [DEBUG] memberlist: Initiating push/pull sync with: 172.21.164.20:8003\n"

looking forward to your reply

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing the alert flow around dispatch.go:135 and the HA activity shown in cluster.go:306, using the supplied Alertmanager v0.20.0 configuration and logs as a reproduction reference. Investigate grouped alerts across two HA instances and verify that one firing alert produces only one resolved notification, including when another alert arrives during the group interval.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.