prometheus / prometheus/alertmanager

Alertmanager pod msg="dropping messages because too many are queued"

Open
#2,440 5 comments 4 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

component/high availability kind/more-info-needed kind/support stale
Dominant language
Go
Stars
8.6k
Forks
2.5k
Avg merge
2d 6h
Merged PRs (30d)
61

Description

hello,

What did you do?
I'am using alertmanager in a kubernetes pod, it's connected to Prometheus, Karma and Kthnxbye to ack alerts.

What did you expect to see?

normal memory usage as before

What did you see instead?

Recently, the memory usage graph of alertmanager is experiencing a linear increase.
In the alertmanager logs, I have this message:
level=warn ts=2020-12-17T09:32:04.281Z caller=delegate.go:272 component=cluster msg="dropping messages because too many are queued" current=4100 limit=4096
Rule expression of the message :

// handleQueueDepth ensures that the queue doesn't grow unbounded by pruning
// older messages at regular interval.
func (d *delegate) handleQueueDepth() {
	for {
		select {
		case <-d.stopc:
			return
		case <-time.After(15 * time.Minute):
			n := d.bcast.NumQueued()
			if n > maxQueueSize {
				level.Warn(d.logger).Log("msg", "dropping messages because too many are queued", "current", n, "limit", maxQueueSize)
				d.bcast.Prune(maxQueueSize)
				d.messagesPruned.Add(float64(n - maxQueueSize))
			}
		}
	}
}

Please note that there is no action to justify this increase.

Environment
Alertmanager : v0.21.0
Prometheus : v2.18.2

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at delegate.go:272 and the handleQueueDepth entry point, then inspect the Alertmanager pod logs and memory graph alongside the reported queue values. Determine why messages are accumulating despite the reported setup; done means a confirmed cause and a scoped fix or clear explanation of the expected queue behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
distributed-systems, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.