hashicorp / hashicorp/consul

Connect Service Mesh does not work with prepared query across datacenters

Open
#8,011 8 comments 12 reactions 0 assignees View on GitHub
theme/connect theme/federation-usability theme/kubernetes theme/prepared-query
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

I am currently using Consul in two different Kubernetes clusters which are connected via [federation](https://www.consul.io/docs/k8s/installation/multi-cluster/kubernetes). I am leveraging both the connect service mesh and prepared queries for connectivity across datacenters.

I wrote a simple proof of concept application which attempts to connect to an upstream in a different Kubernetes cluster (and therefore, a different Consul datacenter). I have found that I cannot make requests to upstream applications in a different datacenter if I am connecting using the Connect Service Mesh with a prepared query. See reproduction steps for details.

#### Reproduction Steps

1. Connect 2 Kubernetes clusters via [federation](https://www.consul.io/docs/k8s/installation/multi-cluster/kubernetes)

2. On each Kubernetes cluster's consul instance(s), ensure the following prepared query is installed to support [geo failover](https://learn.hashicorp.com/consul/developer-discovery/geo-failover#prepared-query-template):

```
curl http://127.0.0.1:8500/v1/query \
--request POST \
--data @- << EOF
{
"Name": "",
"Template": {
"Type": "name_prefix_match"
},
"Service": {
"Service": "${name.full}",
"Failover": {
"NearestN": 2
}
}
}
EOF
```

3. Create 2 Kubernetes applications (service-a and service-b) and deploy them to each Kubernetes cluster. Ensure the pods associated with these applications have the `'consul.hashicorp.com/connect-inject': 'true'` annotation applied. Also, on service-a add the following annotation to enable connecting to service-b in a [variety of supported ways](https://www.consul.io/docs/k8s/connect#consul-hashicorp-com-connect-service-upstreams):

```
'consul.hashicorp.com/connect-service-upstreams': 'service-b:3001,service-b:3002:other-datacenter,prepared_query:service-b:3003'
```

4. If you connect to the pod of service-a you will find you can curl localhost:3001 if service-b is running on the same Kubernetes cluster. If not, this calls fails (and this is expected).

Similarly, you can curl localhost:3002 if service-b is running on the other datacenter/Kubernetes cluster. If not, this call fails (again, I'd expect this behavior).

However, performing a curl against localhost:3003 _only_ works if service-b is running on the local datacenter/Kubernetes cluster. If service-b is running on the non-local datacenter but not on the local datacenter, a call to localhost:3003 fails with an `ECONNRESET` error.

What makes this more puzzling is the fact that if I connect to the pod of service-a I see that performing `nslookup service-b.query.consul` returns the IP address of the pod associated with service-b in the other datacenter. This means that the prepared query is working as I'd expect in terms of DNS resolution, but it doesn't seem to work with the connect service mesh.

### Consul info
We are using version 1.8.0-beta2 of Consul.

If there is additional information I can provide that would be helpful in debugging this issue, please let me know!

Contributor guide

Open the contributing guide

Research direction

Reproduce the issue with two federated Kubernetes clusters, the geo-failover prepared query, and the three annotated upstream forms. Start by comparing DNS resolution and Connect behavior for localhost:3003 when service-b is only in the remote datacenter; done means the prepared-query upstream can reach that remote service without ECONNRESET.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.