Schedule deadlocks after Workflow ID reuse when previous scheduled action has Workflow Retry chain

Temporal version

• Temporal Server: 1.31.0
• Persistence: PostgreSQL
• Visibility: Elasticsearch

Summary

A Schedule's internal scheduler Workflow can become permanently blocked in refreshWorkflows when:

1. A Schedule-started Workflow has a Workflow-level Retry Policy.
2. The scheduled execution exhausts retries, producing a multi-run chain ending in failure.
3. A separate manual execution reuses the Schedule-generated Workflow ID that has a multi-run chain.
4. The next scheduled action attempts to refresh the previous action.

The scheduler watcher repeatedly fails with:

WatchWorkflow: last event did not have correct attrs

The Schedule stops launching actions. Pause/unpause signals do not recover it because the scheduler Workflow remains blocked waiting for an endlessly retried local activity.

Reproduction

Create a Schedule like:

ScheduleActionStartWorkflow{
    ID:       "example",
    Workflow: ExampleWorkflow,
    RetryPolicy: &temporal.RetryPolicy{
        MaximumAttempts: 3,
    },
}

Use default Skip overlap policy. The Schedule generates a Workflow ID like:

example-2026-06-03T12:20:00Z

Steps:

1. Make the scheduled Workflow fail through all three Workflow retry attempts.

2. Before the next scheduled tick, manually start a separate Workflow execution using the exact generated ID:

   example-2026-06-03T12:20:00Z
   

3. Wait for the next scheduled tick.

Expected behavior

The scheduler identifies the previous scheduled chain as closed, records terminal status, and starts/skips the next action normally.

Actual behavior

The Schedule becomes permanently stuck:

recent action status: RUNNING
actual scheduled Workflow chain status: FAILED
RunningWorkflows: []
bufferSize: 1
next action time: in past

Internal scheduler stack:

refreshWorkflows
processWatcherResult
WatchWorkflow

Internal scheduler history repeatedly records:

WatchWorkflow: last event did not have correct attrs

Pause/unpause signals are accepted into history but are not processed.

Root cause analysis

Scheduled retry chain:

Run A — Continue-As-New
Run B — Continue-As-New
Run C — Failed

Scheduler tracks:

WorkflowId: example-2026-06-03T12:20:00Z
FirstExecutionRunId: Run A

Manual execution creates a newer independent chain with the same Workflow ID:

Run D — separate FirstExecutionRunId

Watcher first queries latest execution by Workflow ID without Run ID:

Execution: &commonpb.WorkflowExecution{WorkflowId: ex.WorkflowId},
FirstExecutionRunId: ex.RunId,

Latest execution is manual Run D. Watcher detects a different chain and follows the scheduled chain using its first Run ID:

if pollRes.FirstExecutionRunId != req.FirstExecutionRunId {
    if len(req.Execution.RunId) == 0 {
        return nil, errFollow(req.FirstExecutionRunId)
    }
}

The resulting response combines:

WorkflowStatus: FAILED
Close event: WORKFLOW_EXECUTION_CONTINUED_AS_NEW

Status represents the terminal retry chain, while the close event belongs to the initial run.

responseBuilder.Build branches on FAILED, expects failed-event attributes, receives a Continue-As-New event, then returns errNoAttrs:

case enumspb.WORKFLOW_EXECUTION_STATUS_FAILED:
    if attrs := event.GetWorkflowExecutionFailedEventAttributes(); attrs == nil {
        return nil, errNoAttrs
    }

The local activity retries forever, blocking the scheduler Workflow.

Relevant source

• Wrong-chain fallback: https://github.com/temporalio/temporal/blob/v1.31.0/service/worker/scheduler/activities.go#L143-L151
• Close-event interpretation: https://github.com/temporalio/temporal/blob/v1.31.0/service/worker/scheduler/activities.go#L302-L349
• Blocking refresh: https://github.com/temporalio/temporal/blob/v1.31.0/service/worker/scheduler/workflow.go#L1544-L1558

Suggested fix

When following FirstExecutionRunId, ensure PollMutableState status and fetched close event refer to the same terminal execution.

Also make errNoAttrs non-retryable or otherwise prevent WatchWorkflow from permanently blocking Schedule processing.