temporalio / temporalio/ui

ui-server: persistent gRPC connection to frontend has no keepalive, silently goes stale after backend pod recycle

Open
#3,824 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
431
Forks
179
Avg merge
2d 9h
Merged PRs (30d)
71

Description

Describe the bug

server/rpc/rpc.go's Dial() creates a single, long-lived *grpc.ClientConn for the lifetime of the ui-server process (this is intentional - see #2149, which moved away from connection-per-request for performance). However, that persistent connection is created with grpc.WithConnectParams (governing reconnect backoff after a failure is already detected) but no grpc.WithKeepaliveParams. Without keepalive pings, the client has no active way to detect a half-dead connection - e.g. if the specific backend Temporal frontend server it originally connected to becomes unreachable (a pod restart/rolling deploy behind a Kubernetes Service, or any environment where the frontend process can change independently of ui-server), the client can be left holding a connection that looks open but never proxies further requests to a live backend.

Every backend-proxied API (/api/v1/namespaces, /api/v1/cluster-info, etc.) then returns 503 immediately (fast, not a timeout - consistent with a connection-level failure rather than a hung RPC), and this persists indefinitely: nothing in ui-server proactively closes and re-dials the stale connection. Only a full process restart (forcing a new grpc.Dial() call) recovers it. ui-server's own /healthz only checks the process itself, not backend gRPC connectivity, so this isn't caught by a typical Kubernetes liveness probe either.

To Reproduce

This is an infrastructure-level repro, not a UI click-path:

  1. Run ui-server pointed at a Temporal frontend running behind a Kubernetes Service, in an environment where frontend pods can be replaced independently of the ui-server pod (e.g. a normal rolling deploy of the Temporal server Deployment).
  2. Leave ui-server running. At some point, cause the specific frontend pod/connection it's using to be replaced (a rolling restart of the frontend Deployment is sufficient - this doesn't require anything exotic).
  3. Without restarting ui-server, call any endpoint that proxies to the Temporal gRPC API, e.g. GET /api/v1/namespaces or GET /api/v1/cluster-info.
  4. Observe a 503 returned quickly (single-digit milliseconds), even though the Temporal cluster itself is healthy and reachable (verified independently via a fresh CLI connection to the same frontend).
  5. Restart the ui-server pod/process (no other change). The same requests immediately start succeeding again.

Expected behavior

The persistent gRPC connection should be resilient to the backend server it's talking to becoming unreachable - either via grpc.WithKeepaliveParams(keepalive.ClientParameters{...}) so gRPC actively probes the connection and triggers the existing reconnect/backoff logic on failure, or via some other mechanism that avoids requiring a full ui-server process restart to recover.

Screenshots

N/A - backend/API-level issue, not a rendering bug.

Desktop (please complete the following information):

  • N/A (server-side gRPC client issue, not browser-specific)

Additional context

  • ui-server version: temporalio/ui:2.52.0
  • Relevant code: server/rpc/rpc.go's Dial() - grpc.ConnectParams is configured, grpc.WithKeepaliveParams is not.
  • Confirmed the mTLS client-cert side of things is not the cause: server/rpc/tls.go's certLoader.GetClientCertificate already correctly reloads the client cert from disk per-handshake, so this isn't a stale-cert issue.
  • Reproduced with a long-running ui-server instance whose backend Temporal frontend Deployment had been rolled several times since ui-server last started - every namespace/cluster-info call failed with 503 until ui-server itself was restarted, at which point everything worked immediately with no other change.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in server/rpc/rpc.go at Dial(), comparing the existing grpc.WithConnectParams setup with gRPC keepalive behavior; review server/rpc/tls.go only to keep certificate reload separate. Reproduce the issue by recycling the Temporal frontend while ui-server remains running, then verify /api/v1/namespaces or /api/v1/cluster-info recovers without restarting ui-server.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
api, backend, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
70/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.