Opaqueness not applied to off-cluster destination with enable-external-profiles annotation
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 11.5k
- Forks
- 1.4k
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 53
Description
### What is the issue?
We're running Linekrd stable-2.12.2
Linkerd is configured with:
```
proxy.opaquePorts: 25,587,3306,4444,5432,6379,26379,9300,11211
```
We set `config.linkerd.io/enable-external-profiles: "true"` annotation on application Pods that connect to a MySQL server off-cluster on port 3306 (following the instructions from https://linkerd.io/2.12/features/protocol-detection/#setting-the-enable-external-profiles-annotation)
However, the application is failing to connect to the MySQL server and we see the following errors in linkerd proxy logs:
```
[ 12.990661s] INFO ThreadId(01) outbound:proxy{addr=10.14.0.218:3306}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
```
the address `10.14.0.218` is outside the cluster networks ranges (defined as: `clusterNetworks: 172.20.0.0/17,172.20.128.0/17`)
Here's the manifest metadata of the running Pod:
```
kind: Pod
metadata:
annotations:
checksum/configmap-key-config.properties: ec936facad2bfc7bf8863ae2b8d3f90356bdfc94e2940ed31654f43abb2b0efb
cni.projectcalico.org/containerID: 590f016aabbac75a6825ad52e018ea71e4e3d09d341d8b232d6a17cf200e7eca
cni.projectcalico.org/podIP: 172.20.11.247/32
cni.projectcalico.org/podIPs: 172.20.11.247/32
config.linkerd.io/enable-external-profiles: "true"
linkerd.io/created-by: linkerd/proxy-injector stable-2.12.2
linkerd.io/inject: enabled
linkerd.io/proxy-version: stable-2.12.2
linkerd.io/trust-root-sha256: 1d57b9c015280710eafad0935ee3ec0bc4d7eb430908e89ae20c5ab7e5ec9f80
vault.security.banzaicloud.io/vault-addr: https://vault.vault.svc:8200
vault.security.banzaicloud.io/vault-env-daemon: "false"
vault.security.banzaicloud.io/vault-role: k8s-eventbus-maxwell
viz.linkerd.io/tap-enabled: "true"
```
I was reviewing a related issue https://github.com/linkerd/linkerd2/issues/8273, which seem to suggest that this was fixed by https://github.com/linkerd/linkerd2-proxy/pull/1617 and from what I can tell should be included in stable-2.12.2, unfortunately we are not able to get this to work as expected.
For now we're using `config.linkerd.io/skip-outbound-ports: "3306"` as a workaround, but we are hoping to not need this and use the external profiles method instead.
### How can it be reproduced?
1. Deploy Linkerd stable-2.12.2
2. Run an application Pod with `config.linkerd.io/enable-external-profiles: "true"` annotation connecting to a MySQL server on port 3306 running off-cluster (not in the `clusterNetworks` range(s))
3. Observe as applications fails to connect and linkerd-proxy reports `protocol detection timed out after 10s`
### Logs, error output, etc
```
[ 12.990661s] INFO ThreadId(01) outbound:proxy{addr=10.14.0.218:3306}: linkerd_detect: Continuing after timeout: linkerd_proxy_http::version::Version protocol detection timed out after 10s
```
### output of `linkerd check -o short`
```text
Linkerd core checks
===================
linkerd-version
---------------
‼ cli is up-to-date
is running version 2.12.2 but the latest stable version is 2.12.4
see https://linkerd.io/2.12/checks/#l5d-version-cli for hints
control-plane-version
---------------------
‼ control plane is up-to-date
is running version 2.12.2 but the latest stable version is 2.12.4
see https://linkerd.io/2.12/checks/#l5d-version-control for hints
linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
some proxies are not running the current version:
* linkerd-destination-5cc958f64c-jjbhq (stable-2.12.2)
* linkerd-destination-5cc958f64c-lj8ss (stable-2.12.2)
* linkerd-destination-5cc958f64c-rjmlq (stable-2.12.2)
* linkerd-identity-84f9d7cf87-6jtxc (stable-2.12.2)
* linkerd-identity-84f9d7cf87-g5ndc (stable-2.12.2)
* linkerd-identity-84f9d7cf87-phbjm (stable-2.12.2)
* linkerd-proxy-injector-5cd47b84fd-dxpkg (stable-2.12.2)
* linkerd-proxy-injector-5cd47b84fd-phwcv (stable-2.12.2)
* linkerd-proxy-injector-5cd47b84fd-zkbq2 (stable-2.12.2)
see https://linkerd.io/2.12/checks/#l5d-cp-proxy-version for hints
Linkerd extensions checks
=========================
linkerd-viz
-----------
‼ viz extension proxies are up-to-date
some proxies are not running the current version:
* metrics-api-855d59f76c-68nz9 (stable-2.12.2)
* prometheus-f7c9f5f74-88djq (stable-2.12.2)
* tap-74db455fc9-p4gvh (stable-2.12.2)
* tap-74db455fc9-qvqxt (stable-2.12.2)
* tap-74db455fc9-v92b4 (stable-2.12.2)
* tap-injector-5875b778dc-hfmcx (stable-2.12.2)
* web-576647df96-mnvh6 (stable-2.12.2)
see https://linkerd.io/2.12/checks/#l5d-viz-proxy-cp-version for hints
Status check results are √
```
### Environment
- Kubernetes version: 1.23.10
- Environment: kops 1.25.3
- Host OS: Ubuntu 20.04.5 LTS (Kernel 5.15.0-1021-aws)
- Linkerd version: stable-2.12.2
### Possible solution
as a workaround, we are currently using the `config.linkerd.io/skip-outbound-ports` annotation to skip port 3306 on Pods that need to connect to MySQL database off-cluster
### Additional context
Opaqueness for port 3306 works just fine for MySQL database running in-cluster, so this is only affecting connections to MySQL servers running off-cluster.
### Would you like to work on fixing this bug?
None
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing how the enable-external-profiles annotation, clusterNetworks, and outbound protocol detection interact for off-cluster destinations. Reproduce the MySQL connection on port 3306 and compare it with an in-cluster destination; done means the external connection is treated as opaque without requiring skip-outbound-ports and no protocol-detection timeout occurs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, rust
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100