temporalio / temporalio/sdk-java

Temporal SDK gRPC calls fail with DEADLINE_EXCEEDED after node restart (GraalVM / K8s)

Open
#2,840 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
433
Forks
249
Avg merge
5d 6h
Merged PRs (30d)
26

Description

When a Spring Boot application using the Temporal Java SDK is compiled as a GraalVM native image and deployed in Kubernetes, the application fails to communicate with Temporal after node restarts or pod rescheduling.

The same application works correctly:

  • ✅ On JVM (non-native)
  • ✅ In Docker outside Kubernetes
  • ❌ Fails in Kubernetes when running as GraalVM native image

This suggests a compatibility issue between Temporal Java SDK and GraalVM native runtime, potentially related to:

  • gRPC channel lifecycle
  • DNS resolution / service discovery
  • resource or reflection configuration
  • connection reuse after pod rescheduling
Steps to Reproduce
  1. Install Temporal using official Helm chart - https://github.com/temporalio/helm-charts
  2. Deploy demo application - https://github.com/olegdibrov/temporal-graalvm-k8s
    Install the app:
    helm install control {path/to/chart}

Application logic (executed on startup):

List<DescribeNamespaceResponse> namespaces = workflowClient
    .getWorkflowServiceStubs()
    .blockingStub()
    .listNamespaces(ListNamespacesRequest.newBuilder().build())
    .getNamespacesList();
log.info("Found {} namespaces", namespaces.size());
  1. Restart Kubernetes node OR drain node:
    kubectl drain <node> --ignore-daemonsets
    Observe application startup behavior
Actual Behavior

Application fails to start for 10–30 minutes

Repeated errors:
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline CallOptions was exceeded after 9.999s
Temporal cluster is healthy (all pods ready)
Eventually, the application may recover without restart

Expected Behavior
  • Application should reconnect to Temporal immediately after pod restart
  • listNamespaces should succeed consistently
  • No prolonged unavailability if Temporal cluster is healthy
Important Observations
  • Issue only occurs in GraalVM native image
  • Does NOT reproduce on JVM
  • Does NOT reproduce outside Kubernetes
  • Temporal services are reachable and healthy during failure window

Delay (~10–30 minutes) suggests:

  • stale DNS cache
  • broken gRPC channel reuse
  • or native-image-related networking issue
Environment
  • Temporal SDK: 1.33.0
  • GraalVM: 25
  • Java: 25
  • Spring Boot: 3.5.13
  • Kubernetes: v1.30.5
Logs

Example error:

io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline CallOptions was exceeded after 9.999786125s

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked temporal-graalvm-k8s demo, its Helm installation, and the startup listNamespaces call. Reproduce the failure by draining or restarting a Kubernetes node, then compare the native-image behavior with JVM and non-Kubernetes runs using the reported DEADLINE_EXCEEDED logs. Done means the application reconnects and listNamespaces succeeds promptly while the Temporal cluster is healthy.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java, kubernetes, spring-boot
Domain
backend, cloud, distributed-systems, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.