Temporal cli gives context deadline exceeded. Tctl does not

What are you really trying to do?

Use temporal to connect to the frontend via gRPC.

Describe the bug

Using temporal to talk to the frontend gives context deadline exceeded. Using tctl is fine.

Here it is working with tctl:

Here it is NOT working with temporal:

It is going to the SAME backend FQDN. I am showing env to illustrate that I do not have any TEMPORAL_ env vars set.

Minimal Reproduction

Install temporal like how I did on Kubernetes of 1.33.3-gke.1136000

helm install temporal temporalio/temporal --version 0.65.0

Install istio for ingress. DO NOT add the sidecar injection into the namespace. We do not want the mesh feature. Ingress only.

./istioctl install --set profile=minimal -y

Set up ingress gateway:

helm upgrade --install istio-ingressgateway --namespace istio-system istio-official/gateway --version 1.27.0 --values ingress-gateway.yaml

My ingress-gateway.yaml looks like this:

name: istio-ingressgateway
service:
  loadBalancerIP: $MY_INTERNAL_IP
  annotations:
    cloud.google.com/load-balancer-type: Internal
    networking.gke.io/internal-load-balancer-subnet: internal1
  ports:
  - name: status-port
    port: 15021
    protocol: TCP
    targetPort: 15021
  - name: http2
    port: 80
    protocol: TCP
    targetPort: 80
  - name: https
    port: 443
    protocol: TCP
    targetPort: 443
  - name: temporal
    port: 7233
    protocol: TCP
    targetPort: 7233
autoscaling:
  maxReplicas: 10
  targetCPUUtilizationPercentage: 75
resources:
  requests:
    cpu: 250m
    memory: 512Mi
  limits:
    memory: 1Gi
    cpu: "1"

Once it is up (you will need to have your TLS certs) for Gateway and Virtualservice:

gateways:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: web-temporal-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - web-temporal.mycompany.com
    port:
      name: temporal-mutual
      number: 443
      protocol: HTTPS
    tls:
      credentialName: istio-ingressgateway-tls
      mode: SIMPLE
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: backend-temporal-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - backend-temporal.mycompany.com
    port:
      name: backend-temporal-mutual
      number: 7233
      protocol: HTTPS
    tls:
      credentialName: istio-ingressgateway-tls
      mode: MUTUAL

Environment/Versions

• OS and processor: M1 Mac 15.6.1
• Temporal Version: 1.28.1
• Using Kubernetes

Additional context

tctl version 1.18.0
temporal version 1.4.1 (Server 1.28.0, UI 2.39.0)

Start by reproducing the connection with the temporal CLI and tctl against the same backend FQDN, using the Kubernetes, Helm, and Istio setup described in the issue. Compare their gRPC and TLS behavior; done means temporal connects to the frontend without returning context deadline exceeded.