ui-server: persistent gRPC connection to frontend has no keepalive, silently goes stale after backend pod recycle
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 431
- Forks
- 179
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 71
Description
Describe the bug
server/rpc/rpc.go's Dial() creates a single, long-lived *grpc.ClientConn for the lifetime of the ui-server process (this is intentional - see #2149, which moved away from connection-per-request for performance). However, that persistent connection is created with grpc.WithConnectParams (governing reconnect backoff after a failure is already detected) but no grpc.WithKeepaliveParams. Without keepalive pings, the client has no active way to detect a half-dead connection - e.g. if the specific backend Temporal frontend server it originally connected to becomes unreachable (a pod restart/rolling deploy behind a Kubernetes Service, or any environment where the frontend process can change independently of ui-server), the client can be left holding a connection that looks open but never proxies further requests to a live backend.
Every backend-proxied API (/api/v1/namespaces, /api/v1/cluster-info, etc.) then returns 503 immediately (fast, not a timeout - consistent with a connection-level failure rather than a hung RPC), and this persists indefinitely: nothing in ui-server proactively closes and re-dials the stale connection. Only a full process restart (forcing a new grpc.Dial() call) recovers it. ui-server's own /healthz only checks the process itself, not backend gRPC connectivity, so this isn't caught by a typical Kubernetes liveness probe either.
To Reproduce
This is an infrastructure-level repro, not a UI click-path:
- Run ui-server pointed at a Temporal frontend running behind a Kubernetes Service, in an environment where frontend pods can be replaced independently of the ui-server pod (e.g. a normal rolling deploy of the Temporal server Deployment).
- Leave ui-server running. At some point, cause the specific frontend pod/connection it's using to be replaced (a rolling restart of the frontend Deployment is sufficient - this doesn't require anything exotic).
- Without restarting ui-server, call any endpoint that proxies to the Temporal gRPC API, e.g.
GET /api/v1/namespacesorGET /api/v1/cluster-info. - Observe a
503returned quickly (single-digit milliseconds), even though the Temporal cluster itself is healthy and reachable (verified independently via a fresh CLI connection to the same frontend). - Restart the ui-server pod/process (no other change). The same requests immediately start succeeding again.
Expected behavior
The persistent gRPC connection should be resilient to the backend server it's talking to becoming unreachable - either via grpc.WithKeepaliveParams(keepalive.ClientParameters{...}) so gRPC actively probes the connection and triggers the existing reconnect/backoff logic on failure, or via some other mechanism that avoids requiring a full ui-server process restart to recover.
Screenshots
N/A - backend/API-level issue, not a rendering bug.
Desktop (please complete the following information):
- N/A (server-side gRPC client issue, not browser-specific)
Additional context
- ui-server version:
temporalio/ui:2.52.0 - Relevant code:
server/rpc/rpc.go'sDial()-grpc.ConnectParamsis configured,grpc.WithKeepaliveParamsis not. - Confirmed the mTLS client-cert side of things is not the cause:
server/rpc/tls.go'scertLoader.GetClientCertificatealready correctly reloads the client cert from disk per-handshake, so this isn't a stale-cert issue. - Reproduced with a long-running ui-server instance whose backend Temporal frontend Deployment had been rolled several times since ui-server last started - every namespace/cluster-info call failed with 503 until ui-server itself was restarted, at which point everything worked immediately with no other change.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in server/rpc/rpc.go at Dial(), comparing the existing grpc.WithConnectParams setup with gRPC keepalive behavior; review server/rpc/tls.go only to keep certificate reload separate. Reproduce the issue by recycling the Temporal frontend while ui-server remains running, then verify /api/v1/namespaces or /api/v1/cluster-info recovers without restarting ui-server.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- api, backend, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 70/100