temporalio / temporalio/temporal

A workflow task following ShutdownWorker should not time out if no workers are available

Open
#7,566 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Expected Behavior

When a Workflow Task (WFT) is scheduled, it will wait indefinitely (modulus workflow timeout parameters) for a worker to come online without timing itself out. This is working as expected when the workflow has no sticky worker set.

Actual Behavior

After and despite ShutdownWorker being called to indicate graceful worker shutdowns, if no workers are available at the time of WFT scheduling (such as after a timer firing event), a WFT failure with the now-shutdown sticky queue may be observed. After the WFT failure, the workflow will show a WFT being scheduled on the normal queue, which will wait indefinitely for a worker to become available, as usual.

The bug here is mostly cosmetic, as if there are other workers available on the normal queue, they will immediately receive the pending WFT. Still, a WFT failure should not be observable in a workflow's history if all workers are shut down gracefully.

Steps to Reproduce the Problem

  1. Start a single worker for a target task queue.
  2. Start a workflow that creates a timer firing after >10s.
  3. Kill the worker after the workflow's "Timer Started" event has been written (but before the timer fires event).
  4. Observe that, following the "Timer Fired" event, a failed WFT that appears to have been destined for the shutdown sticky queue will be observable in workflow history.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing workflow-task scheduling around ShutdownWorker, sticky queues, and the Timer Fired event. Reproduce the single-worker shutdown scenario described in the issue and inspect the resulting workflow history. Done means graceful shutdown does not produce an observable failed workflow task when no workers are available.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.