Consul catalog and sidecars cannot access clients when deployed with Cilium on EKS

#### Overview of the Issue

Starting from a system where consul server and clients work, after deploying Cilium CNI by following their [instructions](https://docs.cilium.io/en/v1.12/gettingstarted/k8s-install-helm/) including deleting the aws-node CNI, after the Cilium deployment the clients are unreachable with a connection reset issue after the clients restart. The servers continue to work as expected.

#### Reproduction Steps

Steps to reproduce this issue:

1. Create a cluster in EKS with
a. dedicated nodes for consul using taints and nodeAffinities
b. General purpose nodes
2. Deploy consul with the following:
````
syncCatalog:
enabled: true

connectInject:
enabled: true
metrics:
defaultEnableMerging: true
envoyExtraArgs: "--component-log-level upstream:debug,http:debug,router:debug,config:debug"

controller:
enabled: true

ingressGateways:
enabled: false

dns:
enabled: true
````
3. Deploy services that require connect inject sidecars and services that don't require them
4. Delete the aws-node daemonset
5. Deploy Cilium v1.12 using [helm](https://docs.cilium.io/en/v1.12/gettingstarted/k8s-install-helm/)
6. Restart the catalog and a pod that has the sidecars

The catalog and pod will fail to spin up again with the errors noted below

SSHing into a pod with the client deployed and attempting to hit the client results in connection reset. The same command works before cilium is deployed

````
root@:/app# curl 10.100.1.150:8500/v1/catalog/services
curl: (56) Recv failure: Connection reset by peer
````

### Consul info for both Client and Server

Client info

```
output from client 'consul info' command here
```

````
/ $ consul info
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 0e046bbb
version = 1.13.2
version_metadata =
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 2
goroutines = 56
max_procs = 2
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 113
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 4780
members = 5
query_queue = 0
query_time = 1
````

Server info

```
output from server 'consul info' command here
```
````
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 0e046bbb
version = 1.13.2
version_metadata =
consul:
acl = disabled
bootstrap = false
known_datacenters = 1
leader = false
leader_addr = 10.1.3.149:8300
server = true
raft:
applied_index = 56604
commit_index = 56604
fsm_pending = 0
last_contact = 73.149906ms
last_log_index = 56604
last_log_term = 125
last_snapshot_index = 49227
last_snapshot_term = 30
latest_configuration = [{Suffrage:Voter ID:e030fb60-b4ba-5003-5ba3-454c8c56b79a Address:10.1.3.149:8300} {Suffrage:Voter ID:168598f1-05b7-48b9-01c2-77b7854162d7 Address:10.1.1.34:8300} {Suffrage:Voter ID:1536daff-97c8-0dc9-c1c3-f5893ab13a38 Address:10.1.2.34:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 125
runtime:
arch = amd64
cpu_count = 2
goroutines = 124
max_procs = 2
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 113
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 4780
members = 5
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 101
members = 3
query_queue = 0
query_time = 1
````

### Operating system and Environment details

OS, Architecture, and any other information you can provide about the environment.

EKS v1.22 on AWS

Only add-on deployed is the CSI controller, which is ignored in v1.22

Services:

Prometheus with promtail daemonsets
Zookeeper
Kafka

### Log Fragments

Connect Inject init:

````
consul-connect-inject-init 2022-11-01T16:55:18.589Z [ERROR] Unable to get Agent services: error="Get \"http://10.1.1.150:8500/v1/agent/services?filter=Meta%5B%22pod-name%22%5D+%3D%3D+%
22concierge-56c9bb8769-85s4f%22+and+Meta%5B%22k8s-namespace%22%5D+%3D%3D+%22core%22+\": dial tcp 10.1.1.150:8500: connect: connection refused"
consul-connect-inject-init 2022-11-01T16:55:19.590Z [ERROR] Unable to get Agent services: error="Get \"http://10.1.1.150:8500/v1/agent/services?filter=Meta%5B%22pod-name%22%5D+%3D%3D+%
22concierge-56c9bb8769-85s4f%22+and+Meta%5B%22k8s-namespace%22%5D+%3D%3D+%22core%22+\": dial tcp 10.1.1.150:8500: connect: connection refused"
consul-connect-inject-init 2022-11-01T16:55:20.592Z [ERROR] Unable to get Agent services: error="Get \"http://10.1.1.150:8500/v1/agent/services?filter=Meta%5B%22pod-name%22%5D+%3D%3D+%
22concierge-56c9bb8769-85s4f%22+and+Meta%5B%22k8s-namespace%22%5D+%3D%3D+%22core%22+\": dial tcp 10.1.1.150:8500: connect: connection refused"
consul-connect-inject-init 2022-11-01T16:55:21.592Z [ERROR] Unable to get Agent services: error="Get \"http://10.1.1.150:8500/v1/agent/services?filter=Meta%5B%22pod-name%22%5D+%3D%3D+%
22concierge-56c9bb8769-85s4f%22+and+Meta%5B%22k8s-namespace%22%5D+%3D%3D+%22core%22+\": dial tcp 10.1.1.150:8500: connect: connection refused"
consul-connect-inject-init 2022-11-01T16:55:22.593Z [ERROR] Unable to get Agent services: error="Get \"http://10.1.1.150:8500/v1/agent/services?filter=Meta%5B%22pod-name%22%5D+%3D%3D+%
22concierge-56c9bb8769-85s4f%22+and+Meta%5B%22k8s-namespace%22%5D+%3D%3D+%22core%22+\": dial tcp 10.1.1.150:8500: connect: connection refused"
````

Catalog:
````
[GET /health/ready] Error getting leader status: Get "http://10.1.1.150:8500/v1/status/leader": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[GET /health/ready] Error getting leader status: Get "http://10.1.1.150:8500/v1/status/leader": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-11-01T16:51:40.186Z [WARN] to-consul/sink: error registering service: node-name=k8s-sync service-name=kafka-headless-message-queue service="&{ kafka-headless-message-queue-758d81
dcca0a kafka-headless-message-queue [k8s] map[external-k8s-node-name:ip-10-1-3-227.eu-west-2.compute.internal external-k8s-ns:message-queue external-k8s-ref-kind:Pod external-k8s-ref-n
ame:kafka-0 external-source:kubernetes port-tcp-client:9092 port-tcp-internal:9093] 9092 10.1.103.84 map[] {0 0} false 0 0 }" err="Put \"http://10.1.1.150:8500/v1/cata
log/register\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
[GET /health/ready] Error getting leader status: Get "http://10.1.1.150:8500/v1/status/leader": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[GET /health/ready] Error getting leader status: Get "http://10.1.1.150:8500/v1/status/leader": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-11-01T16:51:45.187Z [WARN] to-consul/sink: error registering service: node-name=k8s-sync service-name=kafka-jmx-metrics-message-queue service="&{ kafka-jmx-metrics-message-queue-
a9addca6845e kafka-jmx-metrics-message-queue [k8s] map[external-k8s-node-name:ip-10-1-1-150.eu-west-2.compute.internal external-k8s-ns:message-queue external-k8s-ref-kind:Pod external-
k8s-ref-name:kafka-2 external-source:kubernetes port-http-metrics:5556] 5556 10.1.101.152 map[] {0 0} false 0 0 }" err="Put \"http://10.1.1.150:8500/v1/catalog/registe
r\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
[GET /health/ready] Error getting leader status: Get "http://10.1.1.150:8500/v1/status/leader": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
[GET /health/ready] Error getting leader status: Get "http://10.1.1.150:8500/v1/status/leader": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
2022-11-01T16:51:49.610Z [INFO] terminated received, shutting down
2022-11-01T16:51:49.610Z [INFO] to-consul/sink: stopping service watcher: service-name=cert-manager-webhook-cert-manager service-consul-namespace=""
2022-11-01T16:51:49.610Z [INFO] to-consul/sink: stopping service watcher: service-name=ingress-nginx-controller-admission-ingress-controller service-consul-namespace=""
W1101 16:51:49.610691 1 reflector.go:441] pkg/mod/k8s.io/client-go@v0.22.2/tools/cache/reflector.go:167: watch of *v1.Endpoints ended with: an error on the server ("unable to dec
ode an event from the watch stream: context canceled") has prevented the request from succeeding
````