hashicorp / hashicorp/consul

connect: transparent proxy not dialing directly to target

Open
#12,713 2 comments 0 reactions 0 assignees View on GitHub
theme/connect type/bug
Dominant language
Go
Stars
30.1k
Forks
4.6k
Avg merge
1d 18h
Merged PRs (30d)
39

Description

#### Overview of the Issue

I'm having trouble to get the consul connect transparent proxy running with direct dialing on a k8s `StatefulSet`.

Plain http requests get redirected to a https service.

e.g.:
`curl http://minio-1.minio-svc.minio.svc.cluster.local:9000/export` fails with an empty reply, because the target is a https server.
Calling `curl -k http://minio-1.minio-svc.minio.svc.cluster.local:9000/export` fails, because the https server is configured by consul connect with mTLS and expects a proper TLS certificate.

Though I would expect that `http://minio-1.minio-svc.minio.svc.cluster.local:9000/export` would get resolved to its IP address (e.g. `http://10.244.0.52:9000/export`) and the forwarded by the consul connect envoy proxy to the consul connect target.

#### Reproduction Steps
1. Setup consul cluster with connect and transparent proxy feature.
2. Setup service defaults (in my case via k8s consul CRD `ServiceDefaults`) for the k8s `StatefulSet` with `transparentProxy.dialedDirectly` enabled.
3. Setup k8s `StatefulSet` (e.g. `minio` with 4 replicas) with `consul-connect` enabled and `transparent-proxy` enabled
4. Enter one of the replicas and try dialing via dedicated dns name or directly via IP address an other pod of the same `StatefulSet`

I'm very grateful for any support on this issue.

### Consul info for both Client and Server

Client info

```
output from client 'consul info' command here
```

Server info

```
agent:
check_monitors = 0
check_ttls = 0
checks = 0
services = 0
build:
prerelease =
revision = e319d7ed
version = 1.11.3
consul:
acl = enabled
bootstrap = false
known_datacenters = 1
leader = true
leader_addr = 10.244.0.16:8300
server = true
raft:
applied_index = 26829
commit_index = 26829
fsm_pending = 0
last_contact = 0
last_log_index = 26829
last_log_term = 2
last_snapshot_index = 16492
last_snapshot_term = 2
latest_configuration = [{Suffrage:Voter ID:8b6fc0ad-06b9-783d-7e27-a3931f39dc70 Address:10.244.0.16:8300} {Suffrage:Voter ID:17b752af-4731-01a2-331b-82cc43668f6c Address:10.244.1.22:8300}]
latest_configuration_index = 0
num_peers = 1
protocol_version = 3
protocol_version_max = 3
protocol_version_min = 0
snapshot_version_max = 1
snapshot_version_min = 0
state = Leader
term = 2
runtime:
arch = amd64
cpu_count = 18
goroutines = 336
max_procs = 18
os = linux
version = go1.17.5
serf_lan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 2
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 5
members = 4
query_queue = 0
query_time = 1
serf_wan:
coordinate_resets = 0
encrypted = false
event_queue = 0
event_time = 1
failed = 0
health_score = 0
intent_queue = 0
left = 0
member_time = 2
members = 2
query_queue = 0
query_time = 1
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the transparent proxy setup with a Kubernetes StatefulSet and the ServiceDefaults transparentProxy.dialedDirectly setting, then trace how direct dialing handles DNS names and pod IPs. Use the curl examples and Consul server information in this report to compare the expected direct target connection with the observed HTTPS and mTLS behavior. Done means direct dialing reaches the target without the unexpected redirect or empty reply.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
infrastructure, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.