hashicorp / hashicorp/consul

Requests made through Consul DNS not working inside Kubernetes

Open
#11,045 4 comments 2 reactions 1 assignee Claimed by @blake View on GitHub
theme/connect type/question
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
2d 6h
Merged PRs (30d)
43

Description

#### Overview of the Issue

Hi all,

We have been working on implementing Consul Connect in our infra for the past few weeks and have been running into this issue. We are trying to understand how service to service communication is working and how the envoy proxy is involved. Quick info on our setup:
* Single external Consul server running on a VM outside a kubernetes cluster
* Consul installed in a kubernetes cluster using the official Consul helm chart and transparent proxy enabled
What we experience is when we try to curl from one pod to another, using the services consul DNS name, the request fails. However, if we try Kubernetes internal DNS, the request works fine. Below is an example of running the curl requests from a pod inside the k8s cluster.

```
/ $ curl static-server.default.svc.cluster.local
"hello world"
/ $ curl static-server.service.mesh.consul
curl: (52) Empty reply from server
```

We would expect that the request would work since the traffic is still going to the same pod.

Another issue we ran into is when using https we would get an SSL error indicating that the servers certificate was not trusted. When we manually trusted the servers certificate we would be asked to provide a certificate for mTLS. Our assumption was that with the transparent proxy enabled, the Envoy sidecar would take care of this but it doesn't seem to be the case.

Thanks for looking into this and let me know if I can provide anymore info

#### Reproduction Steps

Steps to reproduce this issue, eg:

1. Create a external consul server and deploy the helm chart below to a Kubernetes cluster
1. Execute a curl request from one pod to another using its Consul DNS name
1. View error

### Consul info for both Client and Server

Client info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 27de64da
version = 1.10.0
consul:
acl = disabled
known_servers = 1
server = false
runtime:
arch = amd64
cpu_count = 4
goroutines = 59
max_procs = 4
os = linux
version = go1.16.5
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 16
failed = 1
health_score = 0
intent_queue = 0
left = 2
member_time = 63016
members = 12
query_queue = 0
query_time = 14
```

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = 27de64da
version = 1.10.0
consul:
acl = enabled
bootstrap = true
known_datacenters = 1
leader = true
leader_addr = 10.x.x.154:8300
server = true
raft:
applied_index = 464760
commit_index = 464760
fsm_pending = 0
last_contact = 0
last_log_index = 464760
last_log_term = 13
last_snapshot_index = 458923
last_snapshot_term = 13
latest_configuration = [{Suffrage:Voter ID:479bbf1c-0cc2-7519-5414-02190ceae24a Address:10.x.x.154:8300}]
latest_configuration_index = 0
num_peers = 0
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 13
runtime:
arch = amd64
cpu_count = 2
goroutines = 475
max_procs = 2
os = linux
version = go1.16.5
serf_lan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 16
failed = 2
health_score = 0
intent_queue = 0
left = 2
member_time = 63009
members = 13
query_queue = 0
query_time = 14
serf_wan:
coordinate_resets = 0
encrypted = true
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 1
members = 1
query_queue = 0
query_time = 1
```

### Operating system and Environment details

Helm chart

```
global:
enabled: false
datacenter: mesh
gossipEncryption:
secretName: 'consul-gossip-encryption-key'
secretKey: key
imageEnvoy: envoyproxy/envoy-alpine:v1.18.4
tls:
enabled: true
enabledAutoEncrypt: true
verify: false
caCert:
secretName: consul-ca-cert
secretKey: tls.crt
caKey:
secretName: consul-ca-key
secretKey: tls.key

client:
enabled: true.
exposeGossipPorts: true
join:
- 'consul-mesh-lb-xxxx.elb.eu-central-1.amazonaws.com'

dns:
enabled: true
clusterIP: 100.71.80.11

ui:
enabled: true
connectInject:
enabled: true
transparentProxy:
defaultEnabled: true
defaultOverwriteProbes: true
controller:
enabled: true

externalServers:
enabled: true
hosts:
- 'consul-mesh-lb-xxx.elb.eu-central-1.amazonaws.com'
httpsPort: 443
tlsServerName: server.mesh.consul
k8sAuthMethodHost: 'https://xxx.eu-central-1.elb.amazonaws.com:443'
```

Consul Server Config

```
{
"advertise_addr": "10.xx.xx.154",
"datacenter": "mesh",
"bootstrap_expect": 1,
"data_dir": "/opt/consul",
"log_level": "INFO",
"enable_syslog": true,
"node_name": "consul-server-mesh-xxx",
"node_meta": {
"ec2_instance_id": "xxx"
},
"acl": {
"enabled": true
},
"server": true,
"connect": {
"enabled": true
},
"addresses": {
"https": "0.0.0.0"
},
"ports": {
"https": 443
},
"rejoin_after_leave": false,
"client_addr": "0.0.0.0",
"encrypt": "xxx",

"primary_datacenter": "mesh",
"telemetry": {
"dogstatsd_addr": "127.0.0.1:8125",
"statsite_address": "127.0.0.1:8125"
}
}
root@consul-mesh:/etc/consul.d# cat tls.json
{
"verify_incoming": false,
"verify_outgoing": false,
"verify_server_hostname": false,
"ca_file": "/etc/ssl/consul-agent-ca.pem",
"cert_file": "/etc/ssl/cert.pem",
"key_file": "/etc/ssl/key.pem",
"auto_encrypt": {
"allow_tls": true,
"dns_san": [
"*.consul"
]
}
}
```

### Log Fragments

Include appropriate Client or Server log fragments. If the log is longer than a few dozen lines, please include the URL to the [gist](https://gist.github.com/) of the log instead of posting it in the issue. Use `-log-level=TRACE` on the client and server to capture the maximum log detail.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.