temporalio / temporalio/sdk-java
Cached activity heartbeat prevents worker from a quick shutdown
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 433
- Forks
- 249
- Avg merge
- 5d 6h
- Merged PRs (30d)
- 26
Description
Expected Behavior
Scheduled throttled activity heartbeats shouldn't delay worker shutdown. The worker shutdown should drain such heartbeats.
Actual Behavior
Worker shutdown is delayed on the shutdown of activity heartbeat executor because there are scheduled heartbeat tasks.
Steps to Reproduce the Problem
See ActivityTimeoutTest#testActivityStartToCloseTimeout that should be taking about 2 seconds, but takes 10 seconds and has
13:39:55.206 [TemporalShutdownManager: 1] WARN i.t.internal.worker.ShutdownManager - Wait for a graceful shutdown of SyncActivityWorker{namespace=default, taskQueue=test-activities, identity=81008@Dmitrys-Temporal-MacBook.local}#heartbeatExecutor timed out, fallback to shutdownNow()
in the shutdown log.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with ActivityTimeoutTest#testActivityStartToCloseTimeout and reproduce its roughly 10-second runtime and shutdown warning. Trace the activity heartbeat executor shutdown path, then verify that scheduled throttled heartbeats are drained so the test completes in about 2 seconds without the graceful-shutdown timeout.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100