[WARN] agent.server.serf.lan: serf: Intent queue depth exceeds limit , dropping messages!
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
Our 5 Consul clusters all emit these messages all the time.
We're using version 1.13.1
This is related to [this](https://github.com/hashicorp/consul/issues/7960) issue.
### Consul info for both Client and Server
Server info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 1
build:
prerelease =
revision = c6d0f9ec
version = 1.13.1
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.200.4.70:8300
server = true
raft:
applied_index = 994304793
commit_index = 994304793
fsm_pending = 0
last_contact = 45.679195ms
last_log_index = 994304793
last_log_term = 419529
last_snapshot_index = 994294563
last_snapshot_term = 419529
latest_configuration = [{Suffrage:Voter ID:0cb59283-962f-9c2a-d850-47dab2a502fc Address:10.200.2.72:8300} {Suffrage:Voter ID:1ca80a4c-6910-8487-6617-24608348ec08 Address:10.200.4.70:8300} {Suffrage:Voter ID:523ec5b0-0454-4e7e-6d0d-47781bd233b6 Address:10.200.10.157:8300} {Suffrage:Voter ID:6024a30e-206e-5f6a-215a-6be0615555c4 Address:10.200.3.166:8300} {Suffrage:Voter ID:54cb2ccb-0e44-f06e-f4ad-d0c6aef3ec29 Address:10.200.1.154:8300}]
latest_configuration_index = 0
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 419529
runtime:
arch = amd64
cpu_count = 8
goroutines = 276620
max_procs = 4
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 23315
failed = 41
health_score = 0
intent_queue = 11318
left = 1395
member_time = 14481495
members = 5456
query_queue = 0
query_time = 394
```
### Operating system and Environment details
Ubuntu 18.04, running on m5d.2xlarge on AWS
### Log Fragments
`2022-08-15T14:29:43.955Z [WARN] agent.server.serf.lan: serf: Intent queue depth (11782) exceeds limit (10966), dropping messages!`
Contributor guide
Research direction
The report names the agent.server.serf.lan warning and related issue #7960, but no source files or tests. Start by reviewing Serf LAN intent-queue handling and reproducing the warning with the supplied Consul 1.13.1 cluster details. Done means identifying the cause and validating a fix or documented mitigation against the reported queue depth.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100