multicluster configuration steps to make NodePort gateway work

Nobody has claimed this yet.

follow up of [6059](https://github.com/linkerd/linkerd2/pull/)

I am testing Linkerd multicluster with NodePort using Linkerd edge-21.8.4 release with two Kubernetes 1.22.1 "west" and "east"

### west/east config
west: master IP 192.168.111.15, worker IP 192.168.111.14, Pod CIDR 11.1.0.0/16
east: master IP 192.168.111.6, worker IP 192.168.111.10, Pod CIDR 12.1.0.0/16

### certificates
```
step certificate create root.linkerd.cluster.local root.crt root.key --profile root-ca --no-password --insecure
step certificate create identity.linkerd.cluster.local issuer.crt issuer.key --profile intermediate-ca --not-after 8760h --no-password --insecure --ca root.crt --ca-key root.key
```

### linkerd install:
```
linkerd install --context west --identity-trust-anchors-file root.crt --identity-issuer-certificate-file issuer.crt --identity-issuer-key-file issuer.key --set clusterNetworks="11.1.0.0/16" | kubectl --context west apply -f -
​​linkerd install --context east --identity-trust-anchors-file root.crt --identity-issuer-certificate-file issuer.crt --identity-issuer-key-file issuer.key --set clusterNetworks="12.1.0.0/16" | kubectl --context east apply -f -
```

### viz install:
```
linkerd viz install \
| tee \
>(kubectl --context=west apply -f -) \
>(kubectl --context=east apply -f -)
```

### multicluster install with NodePort
Ports are 30500 and 30501 (west), 31500 and 31501 (east)
```
linkerd --context=west multicluster install --set gateway.serviceType=NodePort --set gateway.nodePort=30500 --set gateway.probe.nodePort=30501 | kubectl --context=west apply -f -
linkerd --context=east multicluster install --set gateway.serviceType=NodePort --set gateway.nodePort=31500 --set gateway.probe.nodePort=31501 | kubectl --context=east apply -f -
```

### multicluster link setup (created a link on west, referencing east IP/ports)
I am using the east gateway IP of the master (192.168.111.6) and the ports (31500, 31501)
```
linkerd --context=east multicluster link --cluster-name east --gateway-addresses 192.168.111.6 --gateway-port 31500 --set gateway.probe.port=31501 | kubectl --context=west apply -f -
```

### multicluster verify
```
linkerd --context=west multicluster gateways
CLUSTER ALIVE NUM_SVC LATENCY_P50 LATENCY_P95 LATENCY_P99
east True 1 15ms 20ms 20ms
```

### multicluster check
```
linkerd --context=west multicluster check
linkerd-multicluster
--------------------
√ Link CRD exists
√ Link resources are valid
* east
√ remote cluster access credentials are valid
* east
√ clusters share trust anchors
* east
√ service mirror controller has required permissions
* east
√ service mirror controllers are running
* east
√ all gateway mirrors are healthy
* east
√ all mirror services have endpoints
√ all mirror services are part of a Link
√ multicluster extension proxies are healthy
‼ multicluster extension proxies are up-to-date
some proxies are not running the current version:
* linkerd-gateway-55bc6b7777-jjmgb (edge-21.8.4)
* linkerd-service-mirror-east-ffd4448fb-cw28t (edge-21.8.4)
see https://linkerd.io/2/checks/#l5d-multicluster-proxy-cp-version for hints
√ multicluster extension proxies and cli versions match

Status check results are √
```

## Bug Report
For the tests, I am using https://linkerd.io/2.10/tasks/multicluster/#installing-the-test-services.
Services are deployed correctly but remote invocation fails (curl does not get any answer from the remote service, see below)
```

k get svc -A --context west
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 11.2.0.1 443/TCP 9h
kube-system kube-dns ClusterIP 11.2.0.10 53/UDP,53/TCP,9153/TCP 9h
linkerd-multicluster linkerd-gateway NodePort 11.2.251.242 4143:30500/TCP,4191:30501/TCP 9h
linkerd-multicluster probe-gateway-east ClusterIP 11.2.250.160 30501/TCP 9h
linkerd-viz grafana ClusterIP 11.2.59.170 3000/TCP 9h
linkerd-viz metrics-api ClusterIP 11.2.100.102 8085/TCP 9h
linkerd-viz prometheus ClusterIP 11.2.16.178 9090/TCP 9h
linkerd-viz tap ClusterIP 11.2.219.61 8088/TCP,443/TCP 9h
linkerd-viz tap-injector ClusterIP 11.2.22.208 443/TCP 9h
linkerd-viz web ClusterIP 11.2.127.174 8084/TCP,9994/TCP 9h
linkerd linkerd-dst ClusterIP 11.2.25.173 8086/TCP 9h
linkerd linkerd-dst-headless ClusterIP None 8086/TCP 9h
linkerd linkerd-identity ClusterIP 11.2.205.38 8080/TCP 9h
linkerd linkerd-identity-headless ClusterIP None 8080/TCP 9h
linkerd linkerd-policy ClusterIP None 8090/TCP 9h
linkerd linkerd-proxy-injector ClusterIP 11.2.130.113 443/TCP 9h
linkerd linkerd-sp-validator ClusterIP 11.2.3.35 443/TCP 9h
test frontend ClusterIP 11.2.155.118 8080/TCP 9h
test podinfo ClusterIP 11.2.209.205 9898/TCP,9999/TCP 9h
test podinfo-east ClusterIP 11.2.57.121 9898/TCP,9999/TCP 9h

#this is local test on east - works
kubectl --context=east -n test exec -c nginx -it $(kubectl --context=east -n test get po -l app=frontend \
--no-headers -o custom-columns=:.metadata.name) -- /bin/sh -c "curl http://podinfo:9898"

#this is local test on west - works
kubectl --context=west -n test exec -c nginx -it $(kubectl --context=west -n test get po -l app=frontend \
--no-headers -o custom-columns=:.metadata.name) -- /bin/sh -c "curl http://podinfo:9898"

#this is remote invocation test from west to east - not working, empty answer. Same if I use the podinfo-east service IP.
kubectl --context=west -n test exec -c nginx -it $(kubectl --context=west -n test get po -l app=frontend \
--no-headers -o custom-columns=:.metadata.name) -- /bin/sh -c "curl http://podinfo-east:9898"

#same remote invocation test using wget returns
Connecting to podinfo-east:9898 (11.2.142.180:9898)
wget: server returned error: HTTP/1.1 502 Bad Gateway

```

Basically, west does not connect to east. No logs in the linkerd-service-mirror-east/service-mirror Pod
I have also tried to create an nginx deployment, exposed as svc in a test3 ns, same issue
```
kubectl create ns test3 --context east
kubectl create deploy nginx --image=nginx -n test3 --context east
kubectl expose deploy nginx -n test3 --context east --port 80
kubectl --context=east label svc -n test3 nginx mirror.linkerd.io/exported=true
```

the logs in the linkerd-service-mirror-east-ffd4448fb-52hxz -n linkerd-multicluster service-mirror seems fine

```
time="2021-09-28T08:17:44Z" level=info msg="Received: OnUpdateCalled: {svc: Service: {name: nginx, namespace: test3, annotations: [[config.linkerd.io/opaque-ports=25,443,587,3306,4444,5432,6379,9300,11211]], labels [[mirror.linkerd.io/exported=true]]}}" apiAddress="https://192.168.111.6:6443" cluster=remote
time="2021-09-28T08:17:44Z" level=info msg="Received: RemoteServiceCreated: {service: Service: {name: nginx, namespace: test3, annotations: [[config.linkerd.io/opaque-ports=25,443,587,3306,4444,5432,6379,9300,11211]], labels [[mirror.linkerd.io/exported=true]]}}" apiAddress="https://192.168.111.6:6443" cluster=remote
time="2021-09-28T08:17:45Z" level=info msg="Creating a new service mirror for test3/nginx" apiAddress="https://192.168.111.6:6443" cluster=remote
time="2021-09-28T08:17:45Z" level=info msg="Resolved gateway [[{192.168.111.6 nil}]:31500] for test3/nginx" apiAddress="https://192.168.111.6:6443" cluster=remote
time="2021-09-28T08:17:45Z" level=info msg="Creating a new endpoints for test3/nginx" apiAddress="https://192.168.111.6:6443" cluster=remote
time="2021-09-28T08:18:31Z" level=info msg="Received: RepairEndpoints" apiAddress="https://192.168.111.6:6443" cluster=remote
time="2021-09-28T08:19:31Z" level=info msg="Received: RepairEndpoints" apiAddress="https://192.168.111.6:6443" cluster=remote
```
the service mirror is correctly in test3 ns on west but it is not responding
"HTTP request sent, awaiting response... Read error (Connection reset by peer) in headers.)"

```
k get svc -A --context west | grep test3
test3 nginx-east ClusterIP 11.2.179.253 80/TCP 4m11s
```

Am I missing something in the configuration for NodePort multicluster reported above?

Start with the multicluster NodePort install and link commands, then inspect the gateway and service-mirror behavior described in the logs. Reproduce the west-to-east request using the listed NodePort addresses and confirm that the mirrored remote service returns successfully instead of an empty response or HTTP 502.

Written by the indexing model from the issue text.