[WARN] agent.server.serf.lan: serf: Intent queue depth (20358) exceeds limit (4096), dropping messages!

#### Overview of the Issue

I had two decommissioned nodes that needed to be removed from the catalog. To do this, I used the following command:
`consul force-leave `

Doing this for the first node went ok. After I did this for the second node, I got a timeout error (didnt capture the output, sorry). Next up, the logs on my whole cluster got spammed with:

```
Jul 11 18:50:13 consul01 consul[584]: 2023-07-11T18:50:13.333+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20358) exceeds limit (4096), dropping messages!
```

Initially, I restarted consul on the master nodes. Once I did this, the warning did not appear on them. All clients still had the warning, until I manually restarted consul on these nodes.

We are running Consul v1.7.3 on Ubuntu 20.04 x86_64.

---

#### Reproduction Steps

### Consul info for both Client and Server

Note, the output below has been anomized

master consul info:
```
agent:
check_monitors = 0
check_ttls = 2
checks = 6
services = 4
build:
prerelease =
revision = 8b4a3d95
version = 1.7.3
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 172.19.84.104:8300
server = true
raft:
applied_index = 53181077
commit_index = 53181077
fsm_pending = 0
last_contact = 46.346823ms
last_log_index = 53181077
last_log_term = 30682
last_snapshot_index = 53179406
last_snapshot_term = 30680
latest_configuration = [{Suffrage:Voter ID:xxx-4c473ed41b4a Address:172.19.84.103:8300} {Suffrage:Voter ID:yyy-bab3d7796e29 Address:172.19.84.102:8300} {Suffrage:Voter ID:-436ad546b013 Address:172.19.84.104:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 30682
runtime:
arch = amd64
cpu_count = 2
goroutines = 308
max_procs = 2
os = linux
version = go1.13.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 122
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5857
members = 58
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 968
members = 3
query_queue = 0
query_time = 1
```

master config.json:
```
{
"auto_encrypt": {
"allow_tls": true
},
"bind_addr": "0.0.0.0",
"bootstrap_expect": 3,
"ca_file": "/etc/easytls/certs/issuing_ca.cert",
"cert_file": "/etc/easytls/certs/server.cert",
"client_addr": "0.0.0.0",
"data_dir": "/var/consul",
"datacenter": "ka",
"disable_host_node_id": true,
"enable_debug": true,
"encrypt": "some_base64_string",
"encrypt_verify_incoming": true,
"encrypt_verify_outgoing": true,
"key_file": "/etc/easytls/certs/server.key",
"log_level": "info",
"ports": {
"http": -1,
"https": 8501
},
"raft_protocol": 3,
"retry_interval": "15s",
"retry_join": [
"consul01.example.com",
"consul02.example.com",
"consul03.example.com"
],
"server": true,
"telemetry": {
"disable_hostname": true,
"prometheus_retention_time": "60s"
},
"ui": true,
"verify_incoming": true,
"verify_outgoing": true,
"verify_server_hostname": true
}
```

client consul info
```
agent:
check_monitors = 0
check_ttls = 0
checks = 1
services = 1
build:
prerelease =
revision = 8b4a3d95
version = 1.7.3
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 2
goroutines = 57
max_procs = 2
os = linux
version = go1.13.7
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 122
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5857
members = 58
query_queue = 0
query_time = 1
```
client config.json:

```
{
"auto_encrypt": {
"tls": true
},
"bind_addr": "172.19.84.112",
"ca_file": "/etc/ssl/certs/some_environment.pem",
"client_addr": "127.0.0.1",
"data_dir": "/var/consul",
"datacenter": "somedc",
"disable_host_node_id": true,
"enable_debug": false,
"enable_script_checks": true,
"encrypt": "some_base64_string",
"encrypt_verify_incoming": true,
"encrypt_verify_outgoing": true,
"log_level": "info",
"raft_protocol": 3,
"retry_interval": "30s",
"retry_join": [
"172.19.84.102",
"172.19.84.103",
"172.19.84.104"
],
"server": false,
"ui": true,
"verify_incoming": false,
"verify_outgoing": true,
"verify_server_hostname": true
}
```

### Operating system and Environment details

Both masters and clients run Ubuntu 20.04 x86_64. Machines are located in the CEST timezone.

### Log Fragments

```
Jul 11 18:50:43 consul01 consul[584]: 2023-07-11T18:50:43.362+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (21020) exceeds limit (4096), dropping messages!
Jul 11 18:51:13 consul01 consul[584]: 2023-07-11T18:51:13.383+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20180) exceeds limit (4096), dropping messages!
Jul 11 18:51:43 consul01 consul[584]: 2023-07-11T18:51:43.400+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (21142) exceeds limit (4096), dropping messages!
Jul 11 18:52:13 consul01 consul[584]: 2023-07-11T18:52:13.416+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (21249) exceeds limit (4096), dropping messages!
Jul 11 18:52:43 consul01 consul[584]: 2023-07-11T18:52:43.443+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (21114) exceeds limit (4096), dropping messages!
Jul 11 18:53:13 consul01 consul[584]: 2023-07-11T18:53:13.463+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (19912) exceeds limit (4096), dropping messages!
Jul 11 18:53:43 consul01 consul[584]: 2023-07-11T18:53:43.478+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20610) exceeds limit (4096), dropping messages!
Jul 11 18:54:13 consul01 consul[584]: 2023-07-11T18:54:13.498+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (21428) exceeds limit (4096), dropping messages!
Jul 11 18:54:43 consul01 consul[584]: 2023-07-11T18:54:43.516+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20032) exceeds limit (4096), dropping messages!
Jul 11 18:55:13 consul01 consul[584]: 2023-07-11T18:55:13.538+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20292) exceeds limit (4096), dropping messages!
Jul 11 18:55:43 consul01 consul[584]: 2023-07-11T18:55:43.554+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20846) exceeds limit (4096), dropping messages!
Jul 11 18:56:13 consul01 consul[584]: 2023-07-11T18:56:13.576+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (19077) exceeds limit (4096), dropping messages!
Jul 11 18:56:43 consul01 consul[584]: 2023-07-11T18:56:43.594+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20431) exceeds limit (4096), dropping messages!
Jul 11 18:57:13 consul01 consul[584]: 2023-07-11T18:57:13.609+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (19992) exceeds limit (4096), dropping messages!
Jul 11 18:57:43 consul01 consul[584]: 2023-07-11T18:57:43.627+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20708) exceeds limit (4096), dropping messages!
Jul 11 18:58:13 consul01 consul[584]: 2023-07-11T18:58:13.643+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20552) exceeds limit (4096), dropping messages!
Jul 11 18:58:43 consul01 consul[584]: 2023-07-11T18:58:43.660+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20175) exceeds limit (4096), dropping messages!
Jul 11 18:59:13 consul01 consul[584]: 2023-07-11T18:59:13.676+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20196) exceeds limit (4096), dropping messages!
Jul 11 18:59:43 consul01 consul[584]: 2023-07-11T18:59:43.692+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (21954) exceeds limit (4096), dropping messages!
Jul 11 19:00:13 consul01 consul[584]: 2023-07-11T19:00:13.709+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (19286) exceeds limit (4096), dropping messages!
Jul 11 19:00:43 consul01 consul[584]: 2023-07-11T19:00:43.722+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20544) exceeds limit (4096), dropping messages!
Jul 11 19:01:13 consul01 consul[584]: 2023-07-11T19:01:13.737+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (19886) exceeds limit (4096), dropping messages!
Jul 11 19:01:43 consul01 consul[584]: 2023-07-11T19:01:43.752+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (19840) exceeds limit (4096), dropping messages!
Jul 11 19:02:13 consul01 consul[584]: 2023-07-11T19:02:13.770+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (21362) exceeds limit (4096), dropping messages!
Jul 11 19:02:43 consul01 consul[584]: 2023-07-11T19:02:43.793+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20812) exceeds limit (4096), dropping messages!
Jul 11 19:03:13 consul01 consul[584]: 2023-07-11T19:03:13.821+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20426) exceeds limit (4096), dropping messages!
Jul 11 19:03:43 consul01 consul[584]: 2023-07-11T19:03:43.836+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (21099) exceeds limit (4096), dropping messages!
Jul 11 19:04:13 consul01 consul[584]: 2023-07-11T19:04:13.859+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20935) exceeds limit (4096), dropping messages!
Jul 11 19:04:43 consul01 consul[584]: 2023-07-11T19:04:43.876+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (19566) exceeds limit (4096), dropping messages!
Jul 11 19:05:13 consul01 consul[584]: 2023-07-11T19:05:13.890+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20196) exceeds limit (4096), dropping messages!
Jul 11 19:05:43 consul01 consul[584]: 2023-07-11T19:05:43.906+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20304) exceeds limit (4096), dropping messages!
Jul 11 19:06:13 consul01 consul[584]: 2023-07-11T19:06:13.926+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20598) exceeds limit (4096), dropping messages!
Jul 11 19:06:43 consul01 consul[584]: 2023-07-11T19:06:43.946+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20208) exceeds limit (4096), dropping messages!
Jul 11 19:07:13 consul01 consul[584]: 2023-07-11T19:07:13.962+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (21588) exceeds limit (4096), dropping messages!
Jul 11 19:07:43 consul01 consul[584]: 2023-07-11T19:07:43.981+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (19700) exceeds limit (4096), dropping messages!
Jul 11 19:08:13 consul01 consul[584]: 2023-07-11T19:08:13.998+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20556) exceeds limit (4096), dropping messages!
Jul 11 19:08:44 consul01 consul[584]: 2023-07-11T19:08:44.013+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20294) exceeds limit (4096), dropping messages!
Jul 11 19:09:14 consul01 consul[584]: 2023-07-11T19:09:14.027+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (21266) exceeds limit (4096), dropping messages!
Jul 11 19:09:44 consul01 consul[584]: 2023-07-11T19:09:44.043+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20544) exceeds limit (4096), dropping messages!
Jul 11 19:10:14 consul01 consul[584]: 2023-07-11T19:10:14.058+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (19719) exceeds limit (4096), dropping messages!
Jul 11 19:10:44 consul01 consul[584]: 2023-07-11T19:10:44.077+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (20077) exceeds limit (4096), dropping messages!
Jul 11 19:11:14 consul01 consul[584]: 2023-07-11T19:11:14.095+0200 [WARN] agent.server.serf.lan: serf: Intent queue depth (19521) exceeds limit (4096), dropping messages!
```

Start with the `consul force-leave ` entry point and reproduce the timeout using the two-node sequence described in the issue. Trace the Serf LAN intent-queue handling associated with the warning and compare behavior on servers and clients. Done means the reported sequence no longer causes sustained queue growth or repeated intent-queue warnings.

Written by the indexing model from the issue text.