Consul - leader instability - Raft leader not found in server lookup mapping

#### Consul Cluster got into a unstable state

Our consul deployment in a k8s cluster (through helm) - 3 servers. We saw an error from one of the consul-servers-0 that
```
2024-09-16 21:22 agent: Coordinate update error: error="Raft leader not found in server lookup mapping"
```
and then periodic errors (twice or three times a day) after, like this...
```
### From the consul agents
agent.http: Request error: method=GET url=/v1/catalog/services from=192.168.187.228:59522 error="rpc error making call: Raft leader not found in server lookup mapping"
agent.client: RPC failed to server: method=Catalog.ListServices server=192.168.70.185:8300 error="rpc error making call: Raft leader not found in server lookup mapping"

###From the consul servers consul-server-0 and consul-server-1
agent.server: Raft has a leader but other tracking of the node would indicate that the node is unhealthy or does not exist. The network may be misconfigured.: leader=192.168.x.y:8300 (consul-server-2)
```

- the leader did not change during the time of the first error
- there was a user impact in that certain requests did fail

There was no other network maintenance, or issues with cpu/memory etc. Any thoughts on why did this fail, how to recover now (should we restart the leader consul-server-2) and any parameters need be tuned to avoid recurrence?

---

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 0
check_ttls = 39
checks = 39
services = 39
build:
prerelease =
revision = 2c56447e
version = 1.11.1
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 80
goroutines = 125
max_procs = 80
os = linux
version = go1.17.5
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 40
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1020
members = 7
query_queue = 0
query_time = 1

```

```
Client agent HCL config
```

Server info

```
/ $ consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 2c56447e
version = 1.11.1
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 192.168.56.152:8300
server = true
raft:
applied_index = 28764675
commit_index = 28764675
fsm_pending = 0
last_contact = 4.451653ms
last_log_index = 28764675
last_log_term = 172
last_snapshot_index = 28759968
last_snapshot_term = 172
latest_configuration = [{Suffrage:Voter ID:1b113a54-186f-d265-4e56-e6653f61abc6 Address:192.168.187.233:8300} {Suffrage:Voter ID:c266bd28-7448-8a20-18d6-2b075aaf901a Address:192.168.56.152:8300} {Suffrage:Voter ID:71762986-ddd4-e5d5-d147-6175df021bcc Address:192.168.70.185:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 172
runtime:
arch = amd64
cpu_count = 80
goroutines = 200
max_procs = 80
os = linux
version = go1.17.5
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 40
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1020
members = 7
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 407
members = 3
query_queue = 0
query_time = 1
/ $
```

```
Server agent HCL config
```

### Operating system and Environment details

this environment runs on a k8s cluster, v1.24. Consul is version 1.11

### Log Fragments