Intermittent "no healthy upstream" on Ingress gateway exposed apps in a federated kubernetes cluster

#### Overview of the Issue

I am testing consul on VMs and within a Kubernetes Cluster. I deployed the primary dc on VMs (AWS EC2) and the secondary dc on Kubernetes EKS. Federation seems to be working well, no errors in consul monitor -log-level trace.
I exposed Hashicups demo app with ingress gateway on the VM dc, it works well.
I deployed it within the kubernetes cluster and exposed it via ingress gateway and it works intermittently.
The failover from the VM dc to the k8s dc works.

I exposed 2 more applications via ingress gateway in the k8s cluster (simple nginx and httpd) and I've got the same problem.
Every time envoy reconfigures itself, the app becomes unavailable with a "no healthy upstream" message and 503 response.
When envoy has its listener up and running again, it works again. It happens regularly during the day.
The unavailability lasts sometimes almost a minute then its working again.

#### Reproduction Steps

Steps to reproduce this issue, eg:

1. Have a Consul VM provisioned on AWS EC2 with Terraform and hashicups deployed via ingress gateway, mesh gateway enabled.
2. Have a Cluster EKS Ready and deploy Consul with helm, configured to be federated with the VM primary dc
3. Deploy hashicups on the k8s cluster and expose it via ingress gateway
4. Pay attention that the intentions and the ACLs are correct.
5. Refresh regularly the k8s exposed hashicups and check the "no healthy upstream" message.

### Consul info for both Client and Server on VMs

Client info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 2
services = 2
build:
prerelease =
revision = 0e046bbb
version = 1.13.2
version_metadata =
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 2
goroutines = 101
max_procs = 2
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 22
members = 9
query_queue = 0
query_time = 1
```

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 0e046bbb
version = 1.13.2
version_metadata =
consul:
acl = enabled
bootstrap = false
known_datacenters = 2
leader = false
leader_addr = 10.10.24.149:8300
server = true
raft:
applied_index = 536867
commit_index = 536867
fsm_pending = 0
last_contact = 49.743448ms
last_log_index = 536867
last_log_term = 2
last_snapshot_index = 524761
last_snapshot_term = 2
latest_configuration = [{Suffrage:Voter ID:1ccd1a00-0e46-4509-2666-137d186de12f Address:10.10.21.31:8300} {Suffrage:Voter ID:ecf20fcb-5708-885a-f4e6-621bcaa7e573 Address:10.10.24.149:8300} {Suffrage:Voter ID:d334fe59-1a1a-553e-58d2-8d3882e35b59 Address:10.10.22.15:8300}]
latest_configuration_index = 0
num_peers = 2
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Follower
term = 2
runtime:
arch = amd64
cpu_count = 2
goroutines = 239
max_procs = 2
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 22
members = 9
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 34
members = 6
query_queue = 0
query_time = 1
```

### Consul info for both Client and Server on Kubernetes

Client info

```
agent:
check_monitors = 0
check_ttls = 5
checks = 17
services = 12
build:
prerelease =
revision = 0e046bbb
version = 1.13.2
version_metadata =
consul:
acl = disabled
known_servers = 3
server = false
runtime:
arch = amd64
cpu_count = 4
goroutines = 382
max_procs = 4
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 5
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 27
members = 9
query_queue = 0
query_time = 1
```

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 0e046bbb
version = 1.13.2
version_metadata =
consul:
acl = disabled
bootstrap = true
known_datacenters = 2
leader = true
leader_addr = 10.10.21.103:8300
server = true
raft:
applied_index = 258610
commit_index = 258610
fsm_pending = 0
last_contact = 0
last_log_index = 258610
last_log_term = 3
last_snapshot_index = 245952
last_snapshot_term = 3
latest_configuration = [{Suffrage:Voter ID:6af5837e-ba87-6b7f-8b90-f5e7e48a61f1 Address:10.10.21.103:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 3
runtime:
arch = amd64
cpu_count = 4
goroutines = 412
max_procs = 4
os = linux
version = go1.18.1
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 5
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 27
members = 9
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 34
members = 6
query_queue = 0
query_time = 1
```

### Operating system and Environment details

AWS EC2 instances using the ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20220924 ami.
The machines are m5.large.

AWS EKS with kubernetes version 1.23.
Kubernetes nodes are t3.xlarge, ami amazon-eks-node-1.23-v20220914.

### Log Fragments

Envoy sidecar doing this over and over :

[2022-10-19 12:47:12.612][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-19 12:47:12.612][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
**[2022-10-19 12:47:12.614][1][info][upstream] [source/server/lds_api.cc:82] lds: add/update listener 'public_listener:10.10.22.73:20000'**
[2022-10-19 12:50:05.701][1][info][upstream] [source/common/upstream/cds_api_helper.cc:35] cds: add 1 cluster(s), remove 0 cluster(s)
[2022-10-19 12:50:05.701][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-19 12:50:05.844][1][info][upstream] [source/common/upstream/cds_api_helper.cc:72] cds: added/updated 1 cluster(s), skipped 0 unmodified cluster(s)
[2022-10-19 12:50:20.701][1][warning][config] [source/common/config/grpc_subscription_impl.cc:118] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment
[2022-10-19 12:50:20.703][1][info][upstream] [source/common/upstream/cds_api_helper.cc:35] cds: add 1 cluster(s), remove 0 cluster(s)
[2022-10-19 12:50:20.703][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext Using deprecated option 'envoy.extensions.transport_sockets.tls.v3.CertificateValidationContext.match_subject_alt_names' from file common.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-19 12:50:20.816][1][info][upstream] [source/common/upstream/cds_api_helper.cc:72] cds: added/updated 1 cluster(s), skipped 0 unmodified cluster(s)
[2022-10-19 12:50:35.705][1][warning][config] [source/common/config/grpc_subscription_impl.cc:118] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.endpoint.v3.ClusterLoadAssignment
[2022-10-19 12:50:35.713][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-19 12:50:35.713][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-19 12:50:35.713][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-19 12:50:35.713][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-19 12:50:35.713][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-19 12:50:35.713][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-19 12:50:35.713][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
[2022-10-19 12:50:35.713][1][warning][misc] [source/common/protobuf/message_validator_impl.cc:21] Deprecated field: type envoy.type.matcher.v3.RegexMatcher Using deprecated option 'envoy.type.matcher.v3.RegexMatcher.google_re2' from file regex.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/version_history/version_history for details. If continued use of this field is absolutely necessary, see https://www.envoyproxy.io/docs/envoy/latest/configuration/operations/runtime#using-runtime-overrides-for-deprecated-features for how to apply a temporary and highly discouraged override.
**[2022-10-19 12:50:35.714][1][info][upstream] [source/server/lds_api.cc:82] lds: add/update listener 'public_listener:10.10.22.73:20000'**

Start with the Envoy log entry points in source/server/lds_api.cc and source/common/upstream/cds_api_helper.cc, focusing on the repeated ClusterLoadAssignment gRPC timeouts. Reproduce the federated VM and EKS setup described in the steps and correlate listener, cluster, and endpoint updates with the 503 responses. Done means ingress gateway applications remain available during Envoy reconfiguration.

Written by the indexing model from the issue text.