Consul cataloged services failing after upgrading to Consul 1.14.11 and consul-dataplane 1.1.11

#### Overview of the Issue

[Consul cataloged services failing after upgrading to Consul 1.14.11 and consul-dataplane 1.1.11]

#### Reproduction Steps

I have Consul installed on K8s, after upgrading from Consul App version 1.13.9 to 1.14.0 and enabling Consul Dataplanes, I have noticed issues with cataloged services loosing connections to the Primary DC after a few minutes servers are started up mostly with the error below:

$ consul catalog services --datacenter xyz
Error listing services: Unexpected response code: 500 (rpc error getting client: failed to get conn: read tcp 10.10.0.1:67541->10.10.4.1:6443: read: connection reset by peer)

Usually a restart of the mesh-gateway deployment sometimes restores the connection and consul catalog services get listed, other times it doesn’t work after a restart of the mesh gateway pods.

This was first noticed right after upgrading to Consul 1.14.0 and enabling consul-dataplanes:1.0.0. Now I have upgraded all the way up to Consul 1.14.11 and consul-dataplane 1.1.11, yet seeing the same issues with Consul catalog services.

BTW: consul-clients have been disabled too but it made no difference as this happens when consul-clients enabled.
Also worthy of mention too is that running $consul members -wan does successfully list all consul wan members.

Side note: I have tried this on 3 DC’s and 5DC’s and its same catalog issues.

Steps to reproduce this issue, eg:

1. Create a cluster with n client nodes n and n server nodes
2. bash-4.4$ consul catalog services --datacenter xyz
3. Error listing services: Unexpected response code: 500 (Remote DC has no server currently reachable)
bash-4.4$
4. consul members -wan => lists all members correctly
5. restart mesh gateway => kubectl rollout restart deployment -n consul consul-mesh-gateway
6. bash-4.4$ consul catalog services --datacenter xyz ==> lists cataloged services

-->

### Consul Server
Consul v1.14.11
Revision c0c5688
Build Date 2023-10-31T13:58:53Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

### Consul Client
Consul v1.14.11
Revision c0c5688
Build Date 2023-10-31T13:58:53Z
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

$ consul info -token=xxxxxxxx123444
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = c0c5688
version = 1.14.11
version_metadata =
consul:
acl = enabled
bootstrap = false
known_datacenters = 7
leader = false
leader_addr = 10.10.71.148:8300
server = true
raft:
applied_index = 535374922
commit_index = 535374922
fsm_pending = 0
last_contact = 46.596815ms
last_log_index = 535374922
last_log_term = 13494
last_snapshot_index = 535370990
last_snapshot_term = 13490
latest_configuration = [{Suffrage:Voter ID:2d9c97c3-552e-c
Address:10.245.77.13:8300} {Suffrage:Voter ID:67c0c154-9d5e-3200-166e-dfgf
Address:10.245.77.252:8300} {Suffrage:Voter ID:7781afdf-c084-d294-51d0-ed49bd8f Address:10.245.71.148:8300} {Suffrage:Voter ID:2e0d8d28-8ab0-d35f-c96a-767
Address:10.245.73.32:8300} {Suffrage:Voter ID:3f2b4d6f-fe87-5c4c-e3aa-b8524f79 Address:10.245.71.30:8300}]
latest_configuration_index = 0
num_peers = 4
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 13494
runtime:
arch = amd64
cpu_count = 32
goroutines = 453
max_procs = 32
os = linux
version = go1.22.2
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 221
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 58242
members = 80
query_queue = 0
query_time = 13
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 262190
members = 38
query_queue = 0
query_time = 10

```
Output from client 'consul info' command here
```

```
Client agent HCL config
```

Server info

```
Output from server 'consul info' command here
```

```
Server agent HCL config
```

### Operating system and Environment details

### Log Fragments