temporalio / temporalio/temporal
Support more direct/immediate notification of a cancelled activity
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
We have long-running activities that need to be canceled on occasion. When the need arises, we typically confirm that the activity has been fully canceled before proceeding to another step related to the side effects that this activity may produce, such as performing IO to an external system. Because of this dynamic, the cancellation has real-time synchronous implications on the orchestrating workflow, visible to a human driving a UI.
Because of this real-time dynamic, we use a relatively aggressive heartbeat strategy (currently 5-second heartbeats with 20-second heartbeat timeouts). If the Temporal system delivered the ActivityCancelled exception at the first heartbeat post request, this would probably be acceptable latency. However, things are compounded by the internal heartbeat suppression algorithm such that we only get the notification once we hit the 80% timeout threshold, in this case, around 15-20 seconds.
We could tune the heartbeat/timeout to be more aggressive (e.g., 1/5 seconds). However, this is a lot of overhead for something that may run months at a time and only very rarely needs to be canceled. The feature proposal is to offer some mechanism for delivering cancelation notifications to an Activity. This could be as simple as a call like "isCanceled()" that is not subject to the heartbeat optimizations, to some callback being registered that is invoked by the framework.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are named. Start by locating the activity heartbeat and cancellation handling, then determine how cancellation can be delivered without relying on heartbeat suppression. Done means an activity receives a timely cancellation notification without requiring an aggressively short heartbeat interval.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100