Timer keeps running after workflow cancellation raising an error
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 431
- Forks
- 179
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 71
Description
Describe the bug
Consider this:
@workflow.run
async def run():
try:
status = await workflow.execute_activity(
start_activity,
args=[request_id, team_id, template_id],
start_to_close_timeout=timedelta(seconds=10),
retry_policy=retry_policy,
)
while True:
if status == X:
break
await workflow.sleep(30)
except asyncio.CancelledError:
status = await workflow.execute_activity(
cleanup_activity,
args=[request_id, team_id, template_id],
start_to_close_timeout=timedelta(seconds=10),
retry_policy=retry_policy,
)
if status == X:
raise
await workflow.sleep(30)
The problem I'm dealing with is that when doing await client.get_workflow_handle("your_workflow_id").cancel()
even though the except code is running fine, the first sleep in the try block keeps running and finally showing an error, even though this error do not interrupt the normal flow, in the Temporal UI the clock keeps ticking until it finishes, causing it to display incorrect information since it should have been stoped on cancellation.
Note: I have also tried asyncio.shield() but the error remains
To Reproduce
Steps to reproduce the behavior:
- do
await client.get_workflow_handle("your_workflow_id").cancel() - have a sleep in the try block and another in the except asyncio.CancelledError block, the second sleep should have enough time to allow the first one to finish and show the error
Part of the Error:
ERROR temporalio.worker._workflow_instance Exception in callback _WorkflowInstanceImpl.workflow_sleep..() at
temporal/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py", line 1727, in
lambda: fut.set_result(None),
^^^^^^^^^^^^^^^^^^^^
asyncio.exceptions.InvalidStateError: invalid state
Expected behavior
every await task should stop in the try block including the sleeps when the workflow is cancelled
Screenshots
as seen in the image the first sleep kept running for about 10 more seconds after the cancellation request
Desktop (please complete the following information):
- OS: ubuntu 23
- Browser chrome
- Version 2.39.0
Additional context
Add any other context about the problem here.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the reported _WorkflowInstanceImpl.workflow_sleep callback at temporal/.venv/lib/python3.12/site-packages/temporalio/worker/_workflow_instance.py line 1727, then reproduce cancellation through client.get_workflow_handle("your_workflow_id").cancel(). Compare the sleep in the try block with the cleanup sleep in the CancelledError handler. Done means the original sleep stops without InvalidStateError and the UI no longer shows the timer running after cancellation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, typescript
- Domain
- frontend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100