Workflow task hang on cold start + matching service rejects retries with 'task already started'

Environment

• Temporal Server: 1.31.0 (temporalio/server:1.31.0)
• Deployment: all-in-one Docker Compose (single container, all four services)
• PostgreSQL: DigitalOcean Managed (db-s-1vcpu-1gb) with TLS required
• Connection pools tested: 4/4/2/2 and 10/10/4/4 (core/visibility max/idle)
• Clean install: databases dropped and recreated, schemas applied via setup-schema + update-schema

Summary

After a clean install of Temporal 1.31.0, the first workflow task executed on any system workflow hangs for the full timeout duration (tested at 10s, 60s, 120s, 300s — always consumes the entire timeout). After the timeout, the matching service permanently blocks retry attempts with serviceerror.TaskAlreadyStarted, creating a deadlock where the workflow can never progress.

Affects both the new priority-based matcher (priTaskMatcher) and the legacy matcher (matching.useNewMatcher: false).

Reproduction

1. Clean install Temporal 1.31.0 with PostgreSQL + TLS
2. Wait for server startup to complete
3. Create a namespace: temporal operator namespace create -n test-ns --retention 24h
4. Delete it: temporal operator namespace delete -n test-ns --yes
5. Observe: temporal-sys-delete-namespace-workflow starts, first workflow task dispatches, then hangs

Reproduces 100% of the time on first system workflow task after server start.

Observed Behavior

Phase 1: First task hangs

Workflow history shows task dispatched (WorkflowTaskStarted) but never completes:

ID  Time                     Type
 1  2026-05-19T01:18:26Z     WorkflowExecutionStarted
 2  2026-05-19T01:18:26Z     WorkflowTaskScheduled
 3  2026-05-19T01:18:31Z     WorkflowTaskStarted
 4  2026-05-19T01:18:43Z     WorkflowTaskTimedOut

Task is genuinely blocked, not slow — consumes entire timeout regardless of duration (10s, 60s, 120s, 300s all hang until timeout).

Phase 2: Retries permanently blocked

After timeout, matching service logs (info level, repeats ~every 15s):

{"level":"info","msg":"history client encountered error","service":"matching",
 "error":"Workflow task already started.",
 "service-error-type":"serviceerror.TaskAlreadyStarted"}

Transfer queue processor also fails to deliver retry tasks:

{"level":"warn","msg":"Fail to process task",
 "queue-task-type":"TransferWorkflowTask",
 "error":"context deadline exceeded",
 "error-type":"serviceerror.DeadlineExceeded",
 "attempt":2}

Frontend service also logs "Workflow is busy" on client polls.

Matching service thinks the timed-out task is still in-flight, so no retry can ever be dispatched — permanent deadlock.

Key Observations

Both matchers affected

Tested with:

• New matcher (default in 1.31.0): backlog: "priority" in task queue manager logs → same hang
• Legacy matcher: matching.useNewMatcher: false, backlog: "classic" → identical hang

Both exhibit the same first-task hang and retry blocking.

Connection pools not the cause

Tested with pools at:

• 4/4/2/2 on cold start → first task times out after 10-12s
• 10/10/4/4 on cold start → first task times out after 10-12s

Pool size doesn't affect the hang.

One successful retry (during shard churn)

In a single test run, the retry succeeded after repeated shard re-acquisitions triggered by a separate SQL bug (see related issue). Timeline:

3  00:38:31  WorkflowTaskStarted
4  00:40:31  WorkflowTaskTimedOut      (120s)
5  00:44:36  WorkflowTaskScheduled     (4 min backoff)
6  00:44:39  WorkflowTaskStarted
7  00:44:45  WorkflowTaskCompleted     (6 seconds)

The shard churn likely cleared the stale in-flight task state in the matching service. Once dispatched, the task completed in 6 seconds.

Impact

• temporal operator namespace delete unusable on 1.31.0 all-in-one deployments
• Potentially any system workflow triggered after server cold start
• Scanner workflows (temporal-sys-tq-scanner, temporal-sys-history-scanner) start successfully, suggesting the issue is specific to workflows triggered after initial server startup completes

Workaround

None fully effective. The combination that worked once (by accident, during shard churn):

1. matching.useNewMatcher: false
2. history.defaultWorkflowTaskTimeout: 300s for temporal-system namespace
3. Wait for unrelated errors to trigger shard churn (not controllable)