temporalio / temporalio/sdk-java
Retry transient errors in ActivityCompletionClient.heartbeat
@dplyukhin is already working on this.
Since Aug 11, 2026.
- Dominant language
- Java
- Stars
- 433
- Forks
- 249
- Avg merge
- 5d 6h
- Merged PRs (30d)
- 26
Description
Is your feature request related to a problem? Please describe.
ActivityCompletionClient.heartbeat fails on the first transient error from the server — most often RESOURCE_EXHAUSTED from namespace rate limiting, but also DEADLINE_EXCEEDED and UNAVAILABLE. For async completion, where an external process holds the task token and the heartbeat is the only thing keeping a long-running activity alive, a single rate-limited
heartbeat can cost us the activity via heartbeat timeout.
The worker-side path already handles this. HeartbeatContextImpl.doHeartBeatLocked catches StatusRuntimeException, logs a warning and reschedules after HEARTBEAT_RETRY_WAIT_MILLIS instead of failing the activity.
Describe the solution you'd like
Wrap recordHeartbeat in the same GrpcRetryer the other three reply calls in
ManualActivityCompletionClientImpl already use — complete, fail and reportCancellation
all run inside grpcRetryer.retry(...) with replyGrpcRetryerOptions, and recordHeartbeat is
the only one that doesn't. That gives heartbeat the same RESOURCE_EXHAUSTED handling
(including the message-too-large carve-out) for free.
A shorter expiration than replyGrpcRetryerOptions may make more sense here, since retrying past
the next heartbeat interval is pointless.
Open to another approach if you have one in mind — making the retryability decision public so
callers can apply their own policy would also unblock us.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.