Connect Service Mesh does not work with prepared query across datacenters
- Dominant language
- Go
- Stars
- 30.1k
- Forks
- 4.6k
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 39
Description
#### Overview of the Issue
I am currently using Consul in two different Kubernetes clusters which are connected via [federation](https://www.consul.io/docs/k8s/installation/multi-cluster/kubernetes). I am leveraging both the connect service mesh and prepared queries for connectivity across datacenters.
I wrote a simple proof of concept application which attempts to connect to an upstream in a different Kubernetes cluster (and therefore, a different Consul datacenter). I have found that I cannot make requests to upstream applications in a different datacenter if I am connecting using the Connect Service Mesh with a prepared query. See reproduction steps for details.
#### Reproduction Steps
1. Connect 2 Kubernetes clusters via [federation](https://www.consul.io/docs/k8s/installation/multi-cluster/kubernetes)
2. On each Kubernetes cluster's consul instance(s), ensure the following prepared query is installed to support [geo failover](https://learn.hashicorp.com/consul/developer-discovery/geo-failover#prepared-query-template):
```
curl http://127.0.0.1:8500/v1/query \
--request POST \
--data @- << EOF
{
"Name": "",
"Template": {
"Type": "name_prefix_match"
},
"Service": {
"Service": "${name.full}",
"Failover": {
"NearestN": 2
}
}
}
EOF
```
3. Create 2 Kubernetes applications (service-a and service-b) and deploy them to each Kubernetes cluster. Ensure the pods associated with these applications have the `'consul.hashicorp.com/connect-inject': 'true'` annotation applied. Also, on service-a add the following annotation to enable connecting to service-b in a [variety of supported ways](https://www.consul.io/docs/k8s/connect#consul-hashicorp-com-connect-service-upstreams):
```
'consul.hashicorp.com/connect-service-upstreams': 'service-b:3001,service-b:3002:other-datacenter,prepared_query:service-b:3003'
```
4. If you connect to the pod of service-a you will find you can curl localhost:3001 if service-b is running on the same Kubernetes cluster. If not, this calls fails (and this is expected).
Similarly, you can curl localhost:3002 if service-b is running on the other datacenter/Kubernetes cluster. If not, this call fails (again, I'd expect this behavior).
However, performing a curl against localhost:3003 _only_ works if service-b is running on the local datacenter/Kubernetes cluster. If service-b is running on the non-local datacenter but not on the local datacenter, a call to localhost:3003 fails with an `ECONNRESET` error.
What makes this more puzzling is the fact that if I connect to the pod of service-a I see that performing `nslookup service-b.query.consul` returns the IP address of the pod associated with service-b in the other datacenter. This means that the prepared query is working as I'd expect in terms of DNS resolution, but it doesn't seem to work with the connect service mesh.
### Consul info
We are using version 1.8.0-beta2 of Consul.
If there is additional information I can provide that would be helpful in debugging this issue, please let me know!
Contributor guide
Research direction
Reproduce the issue with two federated Kubernetes clusters, the geo-failover prepared query, and the three annotated upstream forms. Start by comparing DNS resolution and Connect behavior for localhost:3003 when service-b is only in the remote datacenter; done means the prepared-query upstream can reach that remote service without ECONNRESET.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes
- Domain
- distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100