hashicorp / hashicorp/consul

[WARN] agent.server.serf.lan: serf: Intent queue depth exceeds limit , dropping messages!

Open
#14,200 6 comments 0 reactions 0 assignees View on GitHub
theme/operator-usability
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue
Our 5 Consul clusters all emit these messages all the time.
We're using version 1.13.1
This is related to [this](https://github.com/hashicorp/consul/issues/7960) issue.

### Consul info for both Client and Server

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 1
build:
prerelease =
revision = c6d0f9ec
version = 1.13.1
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.200.4.70:8300
server = true
raft:
applied_index = 994304793
commit_index = 994304793
fsm_pending = 0
last_contact = 45.679195ms
last_log_index = 994304793
last_log_term = 419529
last_snapshot_index = 994294563
last_snapshot_term = 419529
latest_configuration = [{Suffrage:Voter ID:0cb59283-962f-9c2a-d850-47dab2a502fc Address:10.200.2.72:8300} {Suffrage:Voter ID:1ca80a4c-6910-8487-6617-24608348ec08 Address:10.200.4.70:8300} {Suffrage:Voter ID:523ec5b0-0454-4e7e-6d0d-47781bd233b6 Address:10.200.10.157:8300} {Suffrage:Voter ID:6024a30e-206e-5f6a-215a-6be0615555c4 Address:10.200.3.166:8300} {Suffrage:Voter ID:54cb2ccb-0e44-f06e-f4ad-d0c6aef3ec29 Address:10.200.1.154:8300}]
latest_configuration_index = 0
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 419529
runtime:
arch = amd64
cpu_count = 8
goroutines = 276620
max_procs = 4
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 23315
failed = 41
health_score = 0
intent_queue = 11318
left = 1395
member_time = 14481495
members = 5456
query_queue = 0
query_time = 394
```

### Operating system and Environment details

Ubuntu 18.04, running on m5d.2xlarge on AWS

### Log Fragments

`2022-08-15T14:29:43.955Z [WARN] agent.server.serf.lan: serf: Intent queue depth (11782) exceeds limit (10966), dropping messages!`

Contributor guide

Open the contributing guide

Research direction

The report names the agent.server.serf.lan warning and related issue #7960, but no source files or tests. Start by reviewing Serf LAN intent-queue handling and reproducing the warning with the supplied Consul 1.13.1 cluster details. Done means identifying the cause and validating a fix or documented mitigation against the reported queue depth.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.