service-resolver failover not working on DC / Mesh Gateway failure
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 2d 6h
- Merged PRs (30d)
- 43
Description
#### Overview of the Issue
In a setup with 3 Consul DCs federated via Mesh Gateway, the configured service-resolver failover does not work in case of a DC / Mesh Gateway outage. It only works as expected for service instance failures.
The setup I used to test looks as follows:
Three Consul DC federated.:
One DC hosting a frontend service
The other 2 DCs are hosting an instance of the backend service
I have tried various permutations of the setup with the frontend deployed on VMs, K8s and backend on VMs, K8s. This does not change the behavior.
service-resolver config:
```
apiVersion: consul.hashicorp.com/v1alpha1
kind: ServiceResolver
metadata:
name: web-be
spec:
failover:
"*":
datacenters:
- azure-vm
- aks-k8s-1
- gke-k8s-1
```
When spinning up and down the service instances in the backend DCs, service failover works as expected.
But when shutting down Mesh Gateway (and servers) in the currently resolved backend DC, the backend service does not fail over to the other backend DC as expected.
`consul members -wan` shows all servers in this DC as failed and the Consul client where the frontend service is running logs the error
`[ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.1.0.7:8300 error="rpc error making call: Remote DC has no server currently reachable"`
When checking the frontend Envoy config_dump, the respective upstream SNI header for reaching the backend service is not being updated.
On K8s when scaling the frontend, newly deployed instances start resolving correctly and fail over to the remaining backend DC, while existing instances do not get updated.
#### Reproduction Steps
Steps to reproduce this issue, eg:
1. Set up 3 Consul DC with respective failover service-resolver
2. Simulate complete DC failure of currently resolved backend DC (Shut down Mesh Gateway and servers)
3. The active proxies for the frontend instances will not failover to the backend service in the other available DC.
### Operating system and Environment details
AKS, GKE, Ubuntu VMs
Consul 1.9.3-ent
Envoy 1.16.0
consul-k8s 0.22.0
### Log Fragments
Frontend Client daemon-set when shutting down remote Mesh Gateway
```
2021-02-18T11:42:27.919Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.2.12:8300 error="rpc error making call: rpc error making call: EOF"
2021-02-18T11:42:27.920Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.0.8:8300 error="rpc error making call: rpc error making call: EOF"
2021-02-18T11:42:27.958Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: EOF"
2021-02-18T11:42:27.959Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.2.12:8300 error="rpc error making call: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
2021-02-18T11:42:27.966Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:48790->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:27.966Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.2.12:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.2.12:49514->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:27.975Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:36532->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:27.975Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
2021-02-18T11:42:30.697Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.2.12:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.2.12:54244->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:30.770Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:37826->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:30.780Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:36542->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:30.788Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.2.12:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.2.12:49570->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:30.797Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:37832->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:31.906Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:36548->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:31.906Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
2021-02-18T11:42:31.915Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.2.12:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.2.12:54274->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:31.923Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:48832->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:31.932Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:36550->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:33.874Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.2.12:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.2.12:54308->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:35.454Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:48858->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:35.925Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:50402->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:35.933Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.2.12:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.2.12:49654->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:35.941Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:37874->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:35.949Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:50404->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:36.492Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.2.12:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.2.12:54346->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:37.111Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:37894->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:38.077Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:36580->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:38.514Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.2.12:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.2.12:49696->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:38.524Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:48894->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:38.533Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:50418->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:38.542Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.2.12:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.2.12:49702->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:39.606Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:48918->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:41.245Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:50424->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:41.806Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.2.12:8300 error="rpc error making call: Remote DC has no server currently reachable"
2021-02-18T11:42:45.732Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:37970->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:45.744Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:50442->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:45.747Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.2.12:8300 error="rpc error making call: Remote DC has no server currently reachable"
2021-02-18T11:42:45.756Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:37976->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:45.911Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:36610->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:47.809Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.2.12:8300 error="rpc error making call: Remote DC has no server currently reachable"
2021-02-18T11:42:47.817Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:37998->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:47.830Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:36620->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:47.911Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.2.12:8300 error="rpc error making call: Remote DC has no server currently reachable"
2021-02-18T11:42:48.781Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:38014->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:48.850Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:50458->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:49.182Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.2.12:8300 error="rpc error making call: Remote DC has no server currently reachable"
2021-02-18T11:42:49.458Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:38028->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:50.218Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:50464->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:50.717Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.2.12:8300 error="rpc error making call: Remote DC has no server currently reachable"
2021-02-18T11:42:55.273Z [ERROR] agent.client: RPC failed to server: method=Internal.ServiceDump server=10.248.1.5:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.1.5:38078->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:55.562Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.0.8:8300 error="rpc error making call: rpc error getting client: failed to get conn: read tcp 10.248.0.8:36652->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:56.208Z [ERROR] agent.client: RPC failed to server: method=Health.ServiceNodes server=10.248.2.12:8300 error="rpc error making call: Remote DC has no server currently reachable"
```
Consul server in frontend DC when shutting down remote Mesh Gateway
```
2021-02-18T11:42:27.974Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:37806->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:28.143Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41396 latency=43.104µs
2021-02-18T11:42:30.769Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:37826->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:30.769Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: rpc error: lead thread didn't get connection"
2021-02-18T11:42:30.796Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:37832->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:31.051Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41422 latency=42.801µs
2021-02-18T11:42:31.578Z [DEBUG] agent.server.memberlist.lan: memberlist: Stream connection from=10.248.2.12:50266
2021-02-18T11:42:31.923Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Health.ServiceNodes error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:48832->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:34.049Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41448 latency=37.748µs
2021-02-18T11:42:34.663Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:48854->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:35.453Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Health.ServiceNodes error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:48858->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:35.477Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send indirect ping: write tcp 10.248.1.5:45572->10.248.2.6:8443: write: broken pipe from=10.8.2.6:8302
2021-02-18T11:42:35.940Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Health.ServiceNodes error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:37874->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:36.622Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:37884->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:36.944Z [DEBUG] agent.server.memberlist.wan: memberlist: Failed ping: consul-server-0.azure-vm (timeout reached)
2021-02-18T11:42:37.049Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41478 latency=46.754µs
2021-02-18T11:42:37.111Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Health.ServiceNodes error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:37894->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:38.523Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:48894->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:38.704Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:37910->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:38.944Z [INFO] agent.server.memberlist.wan: memberlist: Suspect consul-server-0.azure-vm has failed, no acks received
2021-02-18T11:42:39.393Z [DEBUG] agent.router.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="consul-server-2.gke-k8s-1 (Addr: tcp/10.248.1.5:8300) (DC: gke-k8s-1)"
2021-02-18T11:42:39.451Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.0.5:8302: read tcp 10.248.1.5:37926->10.248.1.4:8443: read: connection reset by peer
2021-02-18T11:42:39.606Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Health.ServiceNodes error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:48918->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:40.046Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41520 latency=36.983µs
2021-02-18T11:42:41.138Z [DEBUG] agent: Skipping remote check since it is managed automatically: check=serfHealth
2021-02-18T11:42:41.138Z [DEBUG] agent: Node info in sync
2021-02-18T11:42:43.048Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41538 latency=36.213µs
2021-02-18T11:42:45.021Z [DEBUG] agent.server.memberlist.lan: memberlist: Stream connection from=10.248.0.7:44172
2021-02-18T11:42:45.676Z [DEBUG] agent.server.memberlist.lan: memberlist: Initiating push/pull sync with: gke-k8s-1-cpu-c88d18-con-default-pool-6eb2482c-84qt 10.248.0.7:8301
2021-02-18T11:42:45.731Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Health.ServiceNodes error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:37970->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:45.754Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Health.ServiceNodes error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:37976->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:46.052Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41568 latency=35.718µs
2021-02-18T11:42:47.587Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:37990->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:47.606Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:48984->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:42:47.817Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:37998->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:48.232Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send indirect ping: read tcp 10.248.1.5:38002->10.248.1.4:8443: read: connection reset by peer from=10.8.1.8:8302
2021-02-18T11:42:48.781Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Health.ServiceNodes error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:38014->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:49.050Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41610 latency=38.115µs
2021-02-18T11:42:49.458Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:38028->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:50.944Z [DEBUG] agent.server.memberlist.wan: memberlist: Initiating push/pull sync with: consul-server-0.gke-k8s-1 10.248.0.8:8302
2021-02-18T11:42:51.373Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:38040->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:52.053Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41636 latency=56.514µs
2021-02-18T11:42:53.044Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send gossip to 10.1.0.5:8302: read tcp 10.248.1.5:38060->10.248.1.4:8443: read: connection reset by peer
2021-02-18T11:42:53.952Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send compound ping and suspect message to 10.1.0.5:8302: read tcp 10.248.1.5:49054->10.248.2.6:8443: read: connection reset by peer
2021-02-18T11:42:55.048Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41662 latency=43.052µs
2021-02-18T11:42:55.272Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:38078->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:42:58.046Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41682 latency=46.276µs
2021-02-18T11:42:58.944Z [INFO] agent.server.memberlist.wan: memberlist: Suspect consul-server-0.azure-vm has failed, no acks received
2021-02-18T11:42:59.766Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:49100->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:43:01.049Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41704 latency=34.188µs
2021-02-18T11:43:03.552Z [DEBUG] agent.router.manager: Rebalanced servers, new active server: number_of_servers=3 active_server="consul-server-0.aks-k8s-1 (Addr: tcp/10.8.2.6:8300) (DC: aks-k8s-1)"
2021-02-18T11:43:04.149Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41724 latency=39.379µs
2021-02-18T11:43:04.907Z [INFO] agent.server.memberlist.wan: memberlist: Marking consul-server-0.azure-vm as failed, suspect timeout reached (4 peer confirmations)
2021-02-18T11:43:04.907Z [INFO] agent.server.serf.wan: serf: EventMemberFailed: consul-server-0.azure-vm 10.1.0.5
2021-02-18T11:43:04.907Z [INFO] agent.server: Handled event for server in area: event=member-failed server=consul-server-0.azure-vm area=wan
2021-02-18T11:43:04.915Z [ERROR] agent.server.memberlist.wan: memberlist: Failed to send indirect ping: read tcp 10.248.1.5:38142->10.248.1.4:8443: read: connection reset by peer from=10.8.0.8:8302
2021-02-18T11:43:06.276Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:38152->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:43:07.056Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41746 latency=45.998µs
2021-02-18T11:43:08.263Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:38166->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:43:09.046Z [DEBUG] agent.server.serf.wan: serf: forgoing reconnect for random throttling
2021-02-18T11:43:09.185Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:38180->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:43:10.050Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41776 latency=46.12µs
2021-02-18T11:43:11.409Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Health.ServiceNodes error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:38198->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:43:11.944Z [DEBUG] agent.server.memberlist.wan: memberlist: Failed ping: consul-server-2.aks-k8s-1 (timeout reached)
2021-02-18T11:43:12.908Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Health.ServiceNodes error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:38212->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:43:13.052Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41802 latency=48.664µs
2021-02-18T11:43:13.747Z [DEBUG] agent.server.memberlist.wan: memberlist: Stream connection from=10.248.2.12:58102
2021-02-18T11:43:15.016Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:38230->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:43:15.678Z [DEBUG] agent.server.memberlist.lan: memberlist: Initiating push/pull sync with: gke-k8s-1-cpu-c88d18-con-default-pool-654bf379-2qzq 10.248.1.3:8301
2021-02-18T11:43:16.049Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41826 latency=70.814µs
2021-02-18T11:43:16.944Z [DEBUG] agent.server.memberlist.wan: memberlist: Failed ping: consul-server-2.aks-k8s-1 (timeout reached)
2021-02-18T11:43:18.979Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:49244->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:43:18.993Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:49246->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:43:19.144Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41854 latency=39.795µs
2021-02-18T11:43:19.960Z [DEBUG] agent.server.memberlist.lan: memberlist: Stream connection from=10.248.0.8:45100
2021-02-18T11:43:21.828Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:49264->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:43:22.049Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41866 latency=86.802µs
2021-02-18T11:43:25.052Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41886 latency=42.838µs
2021-02-18T11:43:26.739Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Health.ServiceNodes error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:38304->10.248.1.4:8443: read: connection reset by peer"
2021-02-18T11:43:28.053Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41898 latency=34.82µs
2021-02-18T11:43:28.862Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Internal.ServiceDump error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:49304->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:43:31.048Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41914 latency=48.147µs
2021-02-18T11:43:31.585Z [DEBUG] agent.server.memberlist.lan: memberlist: Stream connection from=10.248.2.12:51130
2021-02-18T11:43:34.049Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41930 latency=46.989µs
2021-02-18T11:43:34.694Z [ERROR] agent.server.rpc: RPC failed to server in DC: server=10.1.0.5:8300 datacenter=azure-vm method=Health.ServiceNodes error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:49334->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:43:37.048Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41938 latency=81.34µs
2021-02-18T11:43:39.047Z [DEBUG] agent.server.serf.wan: serf: forgoing reconnect for random throttling
2021-02-18T11:43:40.052Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41946 latency=50.274µs
2021-02-18T11:43:43.053Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41958 latency=34.626µs
2021-02-18T11:43:45.029Z [DEBUG] agent.server.memberlist.lan: memberlist: Stream connection from=10.248.0.7:44384
2021-02-18T11:43:45.680Z [DEBUG] agent.server.memberlist.lan: memberlist: Initiating push/pull sync with: gke-k8s-1-cpu-c88d18-con-default-pool-654bf379-2qzq 10.248.1.3:8301
2021-02-18T11:43:46.049Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41974 latency=45.837µs
2021-02-18T11:43:46.944Z [DEBUG] agent.server.memberlist.wan: memberlist: Failed ping: consul-server-2.aks-k8s-1 (timeout reached)
2021-02-18T11:43:48.539Z [DEBUG] agent.router.manager: pinging server failed: server="consul-server-0.azure-vm (Addr: tcp/10.1.0.5:8300) (DC: azure-vm)" error="rpc error getting client: failed to get conn: read tcp 10.248.1.5:49386->10.248.2.6:8443: read: connection reset by peer"
2021-02-18T11:43:48.539Z [DEBUG] agent.router.manager: No healthy servers during rebalance, aborting
2021-02-18T11:43:49.144Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:41990 latency=41.068µs
2021-02-18T11:43:50.965Z [DEBUG] agent.server.memberlist.wan: memberlist: Initiating push/pull sync with: consul-server-2.aks-k8s-1 10.248.2.6:8443
2021-02-18T11:43:52.051Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:42002 latency=47.291µs
2021-02-18T11:43:52.600Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=azure-vm
2021-02-18T11:43:52.645Z [DEBUG] agent.server.memberlist.wan: memberlist: Stream connection from=10.248.1.4:33886
2021-02-18T11:43:54.427Z [DEBUG] agent.router.manager: Rebalanced servers, new active server: number_of_servers=1 active_server="consul-server-2 (Addr: tcp/10.248.1.5:8300) (DC: gke-k8s-1)"
2021-02-18T11:43:55.049Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:42018 latency=37.796µs
2021-02-18T11:43:58.050Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:42030 latency=38.45µs
2021-02-18T11:43:58.195Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=azure-vm
2021-02-18T11:43:58.245Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=azure-vm
2021-02-18T11:44:01.049Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:42048 latency=63.691µs
2021-02-18T11:44:01.281Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=azure-vm
2021-02-18T11:44:01.838Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=azure-vm
2021-02-18T11:44:04.048Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:42074 latency=44.959µs
2021-02-18T11:44:05.145Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=azure-vm
2021-02-18T11:44:07.050Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:42088 latency=46.467µs
2021-02-18T11:44:07.158Z [WARN] agent.server.rpc: RPC request to DC is currently failing as no server can be reached: datacenter=azure-vm
2021-02-18T11:44:09.047Z [INFO] agent.server.serf.wan: serf: attempting reconnect to consul-server-0.azure-vm 10.1.0.5:8302
2021-02-18T11:44:09.054Z [DEBUG] agent.server.memberlist.wan: memberlist: Failed to join 10.1.0.5: read tcp 10.248.1.5:38514->10.248.1.4:8443: read: connection reset by peer
2021-02-18T11:44:10.143Z [DEBUG] agent.http: Request finished: method=GET url=/v1/status/leader from=127.0.0.1:42106 latency=46.993µs
```
consul members -wan
```
Node Address Status Type Build Protocol DC Segment
consul-server-0.aks-k8s-1 10.8.2.6:8302 alive server 1.9.3+ent 2 aks-k8s-1
consul-server-0.azure-vm 10.1.0.5:8302 failed server 1.9.3+ent 2 azure-vm
consul-server-0.gke-k8s-1 10.248.0.8:8302 alive server 1.9.3+ent 2 gke-k8s-1
consul-server-1.aks-k8s-1 10.8.1.8:8302 alive server 1.9.3+ent 2 aks-k8s-1
consul-server-1.gke-k8s-1 10.248.2.12:8302 alive server 1.9.3+ent 2 gke-k8s-1
consul-server-2.aks-k8s-1 10.8.0.8:8302 alive server 1.9.3+ent 2 aks-k8s-1
consul-server-2.gke-k8s-1 10.248.1.5:8302 alive server 1.9.3+ent 2 gke-k8s-1
```
Envoy cluster config dump of frontend instance running before DC failure where SNI does not get updated (last_updated time is before failure):
```
{
"version_info": "00000017",
"cluster": {
"@type": "type.googleapis.com/envoy.api.v2.Cluster",
"name": "web-be.default.gke-k8s-1.internal.ad775526-821b-5049-98ac-6a6104e9dd79.consul",
"type": "EDS",
"eds_cluster_config": {
"eds_config": {
"ads": {}
}
},
"connect_timeout": "5s",
"circuit_breakers": {},
"tls_context": {
"common_tls_context": {
"tls_params": {},
"tls_certificates": [
{
"certificate_chain": {
"inline_string": "-----BEGIN CERTIFICATE-----\nMIICSDCCAe6gAwIBAgIBDjAKBggqhkjOPQQDAjAwMS4wLAYDVQQDEyVwcmktZnU5\nczhzcS5jb25zdWwuY2EuYWQ3NzU1MjYuY29uc3VsMB4XDTIxMDIxODExMjgxMVoX\nDTIxMDIyMTExMjgxMVowLDEqMCgGA1UEAxMhd2ViZmUuc3ZjLmRlZmF1bHQuYWQ3\nNzU1MjYuY29uc3VsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEnWZFDu84Jtqa\nYjW3hIXIm94ePnfnelq5xgSuyeb6rPfF8V3Z3FXZ9DfNHIrZlE8Yok+OGt65CQn8\nksTLX4ncHqOB/DCB+TAOBgNVHQ8BAf8EBAMCA7gwHQYDVR0lBBYwFAYIKwYBBQUH\nAwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKQYDVR0OBCIEIIsNoiXGjhcTw4/M\nhCxqjazA2+otvC6MU+8DWGq1ko1OMCsGA1UdIwQkMCKAIMEio6JXD0foegC413mR\n7aNOH/O8Yweg84ZQtguUzzK2MGIGA1UdEQRbMFmGV3NwaWZmZTovL2FkNzc1NTI2\nLTgyMWItNTA0OS05OGFjLTZhNjEwNGU5ZGQ3OS5jb25zdWwvbnMvZGVmYXVsdC9k\nYy9na2UtazhzLTEvc3ZjL3dlYi1mZTAKBggqhkjOPQQDAgNIADBFAiA4CaEdggm7\nV19r4zMSmty1err3eLNWVBTFGwh6H32JmwIhAKL4ayOuKuA1LbYbBLHiIeoGYDTs\nnAJfibUYluCq48i9\n-----END CERTIFICATE-----\n"
},
"private_key": {
"inline_string": "[redacted]"
}
}
],
"validation_context": {
"trusted_ca": {
"inline_string": "-----BEGIN CERTIFICATE-----\nMIICDTCCAbOgAwIBAgIBCjAKBggqhkjOPQQDAjAwMS4wLAYDVQQDEyVwcmktZnU5\nczhzcS5jb25zdWwuY2EuYWQ3NzU1MjYuY29uc3VsMB4XDTIxMDIxODExMjU1NFoX\nDTMxMDIxODExMjU1NFowMDEuMCwGA1UEAxMlcHJpLWZ1OXM4c3EuY29uc3VsLmNh\nLmFkNzc1NTI2LmNvbnN1bDBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABCzUIzGa\naDNqDE1WlnUqgCViJdvaRX8jjFn4mM2dAya8YhAt1cGmkKj31/NxteDk2UOFXJ8n\nJvIwnjyYWCNjoDCjgb0wgbowDgYDVR0PAQH/BAQDAgGGMA8GA1UdEwEB/wQFMAMB\nAf8wKQYDVR0OBCIEIMEio6JXD0foegC413mR7aNOH/O8Yweg84ZQtguUzzK2MCsG\nA1UdIwQkMCKAIMEio6JXD0foegC413mR7aNOH/O8Yweg84ZQtguUzzK2MD8GA1Ud\nEQQ4MDaGNHNwaWZmZTovL2FkNzc1NTI2LTgyMWItNTA0OS05OGFjLTZhNjEwNGU5\nZGQ3OS5jb25zdWwwCgYIKoZIzj0EAwIDSAAwRQIhAOphefH3zZuYd1uIxCS6bKLE\nSLAmeaZ9QLNjn12MltdAAiAKDkj6HX6bsmerRwZZKTIcJ2BRinNcphDho/HPe8+U\n0w==\n-----END CERTIFICATE-----\n"
}
}
},
"sni": "web-be.default.azure-vm.internal.ad775526-821b-5049-98ac-6a6104e9dd79.consul"
},
"outlier_detection": {},
"common_lb_config": {
"healthy_panic_threshold": {}
},
"alt_stat_name": "web-be.default.gke-k8s-1.internal.ad775526-821b-5049-98ac-6a6104e9dd79.consul"
},
"last_updated": "2021-02-18T11:40:51.069Z"
}
]
},
```
Envoy cluster config dump of frontend instance deployed after DC failure with correct SNI:
```
{
"version_info": "00000001",
"cluster": {
"@type": "type.googleapis.com/envoy.api.v2.Cluster",
"name": "web-be.default.gke-k8s-1.internal.ad775526-821b-5049-98ac-6a6104e9dd79.consul",
"type": "EDS",
"eds_cluster_config": {
"eds_config": {
"ads": {}
}
},
"connect_timeout": "5s",
"circuit_breakers": {},
"tls_context": {
"common_tls_context": {
"tls_params": {},
"tls_certificates": [
{
"certificate_chain": {
"inline_string": "-----BEGIN CERTIFICATE-----\nMIICRzCCAe6gAwIBAgIBETAKBggqhkjOPQQDAjAwMS4wLAYDVQQDEyVwcmktZnU5\nczhzcS5jb25zdWwuY2EuYWQ3NzU1MjYuY29uc3VsMB4XDTIxMDIxODExNDczNVoX\nDTIxMDIyMTExNDczNVowLDEqMCgGA1UEAxMhd2ViZmUuc3ZjLmRlZmF1bHQuYWQ3\nNzU1MjYuY29uc3VsMFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEJVspOvCtF0kk\n2q3WArIahH/QEcU0NzT7+ptzD0FByQ0Jt8NDBSlYLenUNk7dhI/jHBnuTVMEitcj\nG446UVBxb6OB/DCB+TAOBgNVHQ8BAf8EBAMCA7gwHQYDVR0lBBYwFAYIKwYBBQUH\nAwIGCCsGAQUFBwMBMAwGA1UdEwEB/wQCMAAwKQYDVR0OBCIEICIX4N3vhSWEMVDi\nfpC6LSEh8b/murFzmh+UieGfV43WMCsGA1UdIwQkMCKAIMEio6JXD0foegC413mR\n7aNOH/O8Yweg84ZQtguUzzK2MGIGA1UdEQRbMFmGV3NwaWZmZTovL2FkNzc1NTI2\nLTgyMWItNTA0OS05OGFjLTZhNjEwNGU5ZGQ3OS5jb25zdWwvbnMvZGVmYXVsdC9k\nYy9na2UtazhzLTEvc3ZjL3dlYi1mZTAKBggqhkjOPQQDAgNHADBEAiAUV1/128gO\nFP/W6HnH/k5gVLcIGQcbZV/EHcKZRbBJzAIgY8Z5bSOpDIz4ABJw8WbpuvIefePo\nGk8JKpppkpEl5KU=\n-----END CERTIFICATE-----\n"
},
"private_key": {
"inline_string": "[redacted]"
}
}
],
"validation_context": {
"trusted_ca": {
"inline_string": "-----BEGIN CERTIFICATE-----\nMIICDTCCAbOgAwIBAgIBCjAKBggqhkjOPQQDAjAwMS4wLAYDVQQDEyVwcmktZnU5\nczhzcS5jb25zdWwuY2EuYWQ3NzU1MjYuY29uc3VsMB4XDTIxMDIxODExMjU1NFoX\nDTMxMDIxODExMjU1NFowMDEuMCwGA1UEAxMlcHJpLWZ1OXM4c3EuY29uc3VsLmNh\nLmFkNzc1NTI2LmNvbnN1bDBZMBMGByqGSM49AgEGCCqGSM49AwEHA0IABCzUIzGa\naDNqDE1WlnUqgCViJdvaRX8jjFn4mM2dAya8YhAt1cGmkKj31/NxteDk2UOFXJ8n\nJvIwnjyYWCNjoDCjgb0wgbowDgYDVR0PAQH/BAQDAgGGMA8GA1UdEwEB/wQFMAMB\nAf8wKQYDVR0OBCIEIMEio6JXD0foegC413mR7aNOH/O8Yweg84ZQtguUzzK2MCsG\nA1UdIwQkMCKAIMEio6JXD0foegC413mR7aNOH/O8Yweg84ZQtguUzzK2MD8GA1Ud\nEQQ4MDaGNHNwaWZmZTovL2FkNzc1NTI2LTgyMWItNTA0OS05OGFjLTZhNjEwNGU5\nZGQ3OS5jb25zdWwwCgYIKoZIzj0EAwIDSAAwRQIhAOphefH3zZuYd1uIxCS6bKLE\nSLAmeaZ9QLNjn12MltdAAiAKDkj6HX6bsmerRwZZKTIcJ2BRinNcphDho/HPe8+U\n0w==\n-----END CERTIFICATE-----\n"
}
}
},
"sni": "web-be.default.aks-k8s-1.internal.ad775526-821b-5049-98ac-6a6104e9dd79.consul"
},
"outlier_detection": {},
"common_lb_config": {
"healthy_panic_threshold": {}
},
"alt_stat_name": "web-be.default.gke-k8s-1.internal.ad775526-821b-5049-98ac-6a6104e9dd79.consul"
},
"last_updated": "2021-02-18T11:48:47.050Z"
}
]
},
```
Contributor guide
Assessment
This issue has not been assessed yet.