linkerd / linkerd/linkerd2

Opaqueness not applied to off-cluster destination with enable-external-profiles annotation

Open
#10,354 10 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

area/egress enhancement pinned priority/P2
Dominant language
Go
Stars
11.5k
Forks
1.4k
Avg merge
3d 22h
Merged PRs (30d)
53

Description

### What is the issue?

We're running Linekrd stable-2.12.2

Linkerd is configured with:
```
proxy.opaquePorts: 25,587,3306,4444,5432,6379,26379,9300,11211
```

We set `config.linkerd.io/enable-external-profiles: "true"` annotation on application Pods that connect to a MySQL server off-cluster on port 3306 (following the instructions from https://linkerd.io/2.12/features/protocol-detection/#setting-the-enable-external-profiles-annotation)

However, the application is failing to connect to the MySQL server and we see the following errors in linkerd proxy logs:
```
[ 12.990661s] INFO ThreadId(01) outbound:proxy{addr=10.14.0.218:3306}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
```

the address `10.14.0.218` is outside the cluster networks ranges (defined as: `clusterNetworks: 172.20.0.0/17,172.20.128.0/17`)

Here's the manifest metadata of the running Pod:
```
kind: Pod
metadata:
annotations:
checksum/configmap-key-config.properties: ec936facad2bfc7bf8863ae2b8d3f90356bdfc94e2940ed31654f43abb2b0efb
cni.projectcalico.org/containerID: 590f016aabbac75a6825ad52e018ea71e4e3d09d341d8b232d6a17cf200e7eca
cni.projectcalico.org/podIP: 172.20.11.247/32
cni.projectcalico.org/podIPs: 172.20.11.247/32
config.linkerd.io/enable-external-profiles: "true"
linkerd.io/created-by: linkerd/proxy-injector stable-2.12.2
linkerd.io/inject: enabled
linkerd.io/proxy-version: stable-2.12.2
linkerd.io/trust-root-sha256: 1d57b9c015280710eafad0935ee3ec0bc4d7eb430908e89ae20c5ab7e5ec9f80
vault.security.banzaicloud.io/vault-addr: https://vault.vault.svc:8200
vault.security.banzaicloud.io/vault-env-daemon: "false"
vault.security.banzaicloud.io/vault-role: k8s-eventbus-maxwell
viz.linkerd.io/tap-enabled: "true"
```

I was reviewing a related issue https://github.com/linkerd/linkerd2/issues/8273, which seem to suggest that this was fixed by https://github.com/linkerd/linkerd2-proxy/pull/1617 and from what I can tell should be included in stable-2.12.2, unfortunately we are not able to get this to work as expected.

For now we're using `config.linkerd.io/skip-outbound-ports: "3306"` as a workaround, but we are hoping to not need this and use the external profiles method instead.

### How can it be reproduced?

1. Deploy Linkerd stable-2.12.2
2. Run an application Pod with `config.linkerd.io/enable-external-profiles: "true"` annotation connecting to a MySQL server on port 3306 running off-cluster (not in the `clusterNetworks` range(s))
3. Observe as applications fails to connect and linkerd-proxy reports `protocol detection timed out after 10s`

### Logs, error output, etc

```
[ 12.990661s] INFO ThreadId(01) outbound:proxy{addr=10.14.0.218:3306}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
```

### output of `linkerd check -o short`

```text
Linkerd core checks
===================

linkerd-version
---------------
‼ cli is up-to-date
is running version 2.12.2 but the latest stable version is 2.12.4
see https://linkerd.io/2.12/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
is running version 2.12.2 but the latest stable version is 2.12.4
see https://linkerd.io/2.12/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
some proxies are not running the current version:
* linkerd-destination-5cc958f64c-jjbhq (stable-2.12.2)
* linkerd-destination-5cc958f64c-lj8ss (stable-2.12.2)
* linkerd-destination-5cc958f64c-rjmlq (stable-2.12.2)
* linkerd-identity-84f9d7cf87-6jtxc (stable-2.12.2)
* linkerd-identity-84f9d7cf87-g5ndc (stable-2.12.2)
* linkerd-identity-84f9d7cf87-phbjm (stable-2.12.2)
* linkerd-proxy-injector-5cd47b84fd-dxpkg (stable-2.12.2)
* linkerd-proxy-injector-5cd47b84fd-phwcv (stable-2.12.2)
* linkerd-proxy-injector-5cd47b84fd-zkbq2 (stable-2.12.2)
see https://linkerd.io/2.12/checks/#l5d-cp-proxy-version for hints

Linkerd extensions checks
=========================

linkerd-viz
-----------
‼ viz extension proxies are up-to-date
some proxies are not running the current version:
* metrics-api-855d59f76c-68nz9 (stable-2.12.2)
* prometheus-f7c9f5f74-88djq (stable-2.12.2)
* tap-74db455fc9-p4gvh (stable-2.12.2)
* tap-74db455fc9-qvqxt (stable-2.12.2)
* tap-74db455fc9-v92b4 (stable-2.12.2)
* tap-injector-5875b778dc-hfmcx (stable-2.12.2)
* web-576647df96-mnvh6 (stable-2.12.2)
see https://linkerd.io/2.12/checks/#l5d-viz-proxy-cp-version for hints

Status check results are √
```

### Environment

- Kubernetes version: 1.23.10
- Environment: kops 1.25.3
- Host OS: Ubuntu 20.04.5 LTS (Kernel 5.15.0-1021-aws)
- Linkerd version: stable-2.12.2

### Possible solution

as a workaround, we are currently using the `config.linkerd.io/skip-outbound-ports` annotation to skip port 3306 on Pods that need to connect to MySQL database off-cluster

### Additional context

Opaqueness for port 3306 works just fine for MySQL database running in-cluster, so this is only affecting connections to MySQL servers running off-cluster.

### Would you like to work on fixing this bug?

None

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing how the enable-external-profiles annotation, clusterNetworks, and outbound protocol detection interact for off-cluster destinations. Reproduce the MySQL connection on port 3306 and compare it with an in-cluster destination; done means the external connection is treated as opaque without requiring skip-outbound-ports and no protocol-detection timeout occurs.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes, rust
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.