temporalio / temporalio/sdk-java
Temporal SDK gRPC calls fail with DEADLINE_EXCEEDED after node restart (GraalVM / K8s)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 433
- Forks
- 249
- Avg merge
- 5d 6h
- Merged PRs (30d)
- 26
Description
When a Spring Boot application using the Temporal Java SDK is compiled as a GraalVM native image and deployed in Kubernetes, the application fails to communicate with Temporal after node restarts or pod rescheduling.
The same application works correctly:
- ✅ On JVM (non-native)
- ✅ In Docker outside Kubernetes
- ❌ Fails in Kubernetes when running as GraalVM native image
This suggests a compatibility issue between Temporal Java SDK and GraalVM native runtime, potentially related to:
- gRPC channel lifecycle
- DNS resolution / service discovery
- resource or reflection configuration
- connection reuse after pod rescheduling
Steps to Reproduce
- Install Temporal using official Helm chart - https://github.com/temporalio/helm-charts
- Deploy demo application - https://github.com/olegdibrov/temporal-graalvm-k8s
Install the app:
helm install control {path/to/chart}
Application logic (executed on startup):
List<DescribeNamespaceResponse> namespaces = workflowClient
.getWorkflowServiceStubs()
.blockingStub()
.listNamespaces(ListNamespacesRequest.newBuilder().build())
.getNamespacesList();
log.info("Found {} namespaces", namespaces.size());
- Restart Kubernetes node OR drain node:
kubectl drain <node> --ignore-daemonsets
Observe application startup behavior
Actual Behavior
Application fails to start for 10–30 minutes
Repeated errors:
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline CallOptions was exceeded after 9.999s
Temporal cluster is healthy (all pods ready)
Eventually, the application may recover without restart
Expected Behavior
- Application should reconnect to Temporal immediately after pod restart
- listNamespaces should succeed consistently
- No prolonged unavailability if Temporal cluster is healthy
Important Observations
- Issue only occurs in GraalVM native image
- Does NOT reproduce on JVM
- Does NOT reproduce outside Kubernetes
- Temporal services are reachable and healthy during failure window
Delay (~10–30 minutes) suggests:
- stale DNS cache
- broken gRPC channel reuse
- or native-image-related networking issue
Environment
- Temporal SDK: 1.33.0
- GraalVM: 25
- Java: 25
- Spring Boot: 3.5.13
- Kubernetes: v1.30.5
Logs
Example error:
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: Deadline CallOptions was exceeded after 9.999786125s
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked temporal-graalvm-k8s demo, its Helm installation, and the startup listNamespaces call. Reproduce the failure by draining or restarting a Kubernetes node, then compare the native-image behavior with JVM and non-Kubernetes runs using the reported DEADLINE_EXCEEDED logs. Done means the application reconnects and listNamespaces succeeds promptly while the Temporal cluster is healthy.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java, kubernetes, spring-boot
- Domain
- backend, cloud, distributed-systems, networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100