Multicluster not working properly with k8s-hosted inlets

Nobody has claimed this yet.

### What is the issue?

I am using linkerd multicluster with inlets, since it was recommended in the docs for cases where you don't have the ability to create public LoadBalancers https://linkerd.io/2.12/tasks/multicluster/#prerequisites

Rather than host a droplet or something like that, which I believe is a more common approach, the inlets are hosted inside k8s. Here's a diagram that sort of shows how it works:

```
┌──────────────────────────────────────────────┐ ┌────────────────────────────────────────┐
│ Hetzner Cluster │ │ │ Home Cluster │
│ │ │ │
│ ┌─────────────┐ ┌─────────────┐ │ │ │ ┌─────────────┐ ┌─────────────┐ │
│ │Inlets Server│<──>│ Ingress │<─┼─────────┼─>│Inlets Client│──┬──>│ Linkerd │ │
│ └─────────────┘ └─────────────┘ │ │ │ └─────────────┘ │ │ Gateway │ │
│ ^ │ │ │ └─────────────┘ │
│ │ │ │ │ │ │
│ ┌─────────────────┐ │ │ │ ┌─────────────┐ │
│ │ Inlets service: │ │ │ │ └──>│ K8s API │ │
│ │4143, 4191, 6443 │ │ │ └─────────────┘ │
│ └─────────────────┘ │ │ │ │
│ ^ │ │ │
│ ┌───────┴───────┐ │ │ │ │
│ │ │ │ │ │
│ │ │ │ │ │ │
│ ┌─────────────┐ ┌─────────────┐ │ │ │
│ │probe-gateway│ │ Mirror │ │ │ │ │
│ │ Service │ │ Service │ │ │ │
│ └─────────────┘ └─────────────┘ │ │ │ │
│ │ │ │
└──────────────────────────────────────────────┘ └────────────────────────────────────────┘
```

This is mostly working, but the problem (I think) is where the multicluster Services are pointing towards the IP address of the Inlets Server's service. I'm not sure if this is *supposed* to work or not, but having a service with another service as its endpoint, seems to result in the connection to the outermost service just timing out every time. This means that out of the box, the mirrors do not work, and the health checks also fail. The connection to the K8s API works just fine, I assume because there's no additional service for it.

```console
$ wget -O - http://linkerd-tunnel-data.inlets.svc.cluster.local:4191/ready
Connecting to linkerd-tunnel-data.inlets.svc.cluster.local:4191 (10.43.107.71:4191)
writing to stdout
ready
- 100% |***************| 6 0:00:00 ETA
written to stdout

$ wget -O - http://probe-gateway-home.linkerd-multicluster.svc.cluster.local:4191/ready
Connecting to probe-gateway-home.linkerd-multicluster.svc:4191 (10.43.20.51:4191)
wget: can't connect to remote host (10.43.20.51): Operation timed out

# Note: probe-gateway-home has the following endpoint, matching linkerd-tunnel-data.inlets.svc
#
# - addresses:
# - ip: 10.43.107.71
# ports:
# - name: mc-probe
# port: 4191
# protocol: TCP
```

### How can it be reproduced?

You can take a look at my Kustomize files here:

[Inlets Server](https://github.com/MacroPower/homelab/tree/ddfd5168d06ad034666e507c22ae3c860791fd74/applications/base/inlets)
[Inlets Client](https://github.com/MacroPower/homelab/tree/ddfd5168d06ad034666e507c22ae3c860791fd74/applications/base/inlets-client)
[Multicluster Link](https://github.com/MacroPower/homelab/blob/ddfd5168d06ad034666e507c22ae3c860791fd74/applications/overlays/hcloud/linkerd-multicluster/link.yaml)

### Logs, error output, etc

There aren't any relevant errors, it really seems to me that the problem occurs before any traffic reaches any of the gateways.

### output of `linkerd check -o short`

Note that this is after implementing what I put under "possible solution".

```text
Linkerd core checks
===================

linkerd-identity
----------------
‼ issuer cert is valid for at least 60 days
issuer certificate will expire on 2023-03-23T22:47:36Z
see https://linkerd.io/2/checks/#l5d-identity-issuer-cert-not-expiring-soon for hints

linkerd-version
---------------
‼ can determine the latest version
Get "https://versioncheck.linkerd.io/version.json?version=edge-23.3.3&uuid=cb175c2b-9ba7-4533-8f6e-c71cb0252d4a&source=cli": x509: certificate signed by unknown authority
see https://linkerd.io/2/checks/#l5d-version-latest for hints
‼ cli is up-to-date
unsupported version channel: edge-23.3.3
see https://linkerd.io/2/checks/#l5d-version-cli for hints

control-plane-version
---------------------
‼ control plane is up-to-date
unsupported version channel: edge-23.2.2
see https://linkerd.io/2/checks/#l5d-version-control for hints
‼ control plane and cli versions match
control plane running edge-23.2.2 but cli running edge-23.3.3
see https://linkerd.io/2/checks/#l5d-version-control for hints

linkerd-control-plane-proxy
---------------------------
‼ control plane proxies are up-to-date
some proxies are not running the current version:
* linkerd-destination-846c7566fd-25qsx (edge-23.2.2)
* linkerd-identity-77bf89bf49-dpwjx (edge-23.2.2)
* linkerd-proxy-injector-756c5d54d5-qfn52 (edge-23.2.2)
see https://linkerd.io/2/checks/#l5d-cp-proxy-version for hints
‼ control plane proxies and cli versions match
linkerd-destination-846c7566fd-25qsx running edge-23.2.2 but cli running edge-23.3.3
see https://linkerd.io/2/checks/#l5d-cp-proxy-cli-version for hints

Linkerd extensions checks
=========================

linkerd-multicluster
--------------------
‼ multicluster extension proxies are up-to-date
Get "https://versioncheck.linkerd.io/version.json?version=edge-23.3.3&uuid=cb175c2b-9ba7-4533-8f6e-c71cb0252d4a&source=cli": x509: certificate signed by unknown authority
see https://linkerd.io/2/checks/#l5d-multicluster-proxy-cp-version for hints
‼ multicluster extension proxies and cli versions match
linkerd-gateway-75cf59bc7d-c2ftc running edge-23.2.2 but cli running edge-23.3.3
see https://linkerd.io/2/checks/#l5d-multicluster-proxy-cli-version for hints

Linkerd extensions checks
=========================

linkerd-viz
-----------
‼ linkerd-viz pods are injected
could not find proxy container for tap-745bbc87bf-92dxh pod
see https://linkerd.io/2/checks/#l5d-viz-pods-injection for hints
‼ viz extension pods are running
container "linkerd-proxy" in pod "tap-745bbc87bf-92dxh" is not ready
see https://linkerd.io/2/checks/#l5d-viz-pods-running for hints
‼ viz extension proxies are up-to-date
Get "https://versioncheck.linkerd.io/version.json?version=edge-23.3.3&uuid=unknown&source=cli": x509: certificate signed by unknown authority
see https://linkerd.io/2/checks/#l5d-viz-proxy-cp-version for hints
‼ viz extension proxies and cli versions match
metrics-api-5d9cb67659-hsg4l running edge-23.2.2 but cli running edge-23.3.3
see https://linkerd.io/2/checks/#l5d-viz-proxy-cli-version for hints

Status check results are √
```

### Environment

- Kubernetes version: v1.25.7+k3s1 containerd://1.6.15-k3s1
- Cluster Environment: k3s on hcloud
- Host OS: openSUSE MicroOS 6.2.6-1-default
- Linkerd version: edge-23.2.2

### Possible solution

I was able to fix the health checks by just changing the `probe-gateway` to be of `type: ExternalName` pointing to the Inlets Server's service. After that, the health checks succeed, and also `linkerd mc check` returns the green checkmark. But, this does not fix any of the mirrored services. I'm not sure how, or if it's even possible, to apply this or a similar fix there. In the very least, ExternalName doesn't allow for remapping the ports as far as I know, which isn't important for the probe but is pretty important for mirrored services.

### Additional context

_No response_

### Would you like to work on fixing this bug?

yes

Start by reproducing the timeout with the linked Inlets Server, Inlets Client, and Multicluster Link Kustomize files, comparing the working Kubernetes API connection with the probe and mirrored-service endpoints. Investigate how the multicluster services resolve and route through the inlets service. Done means health checks and mirrored services work with inlets hosted inside Kubernetes.

Written by the indexing model from the issue text.