temporalio / temporalio/temporal
start_workflow_execution fails with duplicate key error despite TERMINATE_EXISTING policy (possible DB inconsistency)
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 23.2k
- Forks
- 1.9k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 228
Description
Expected Behavior
Starting a workflow with the following parameters should either succeed or terminate the existing one as specified by the conflict policy:
- id_reuse_policy: ALLOW_DUPLICATE
- execution_timeout: 12h
- id_conflict_policy: TERMINATE_EXISTING
Actual Behavior
The gRPC call start_workflow_execution fails after multiple retries with the following error:
WARN temporal_client::retry: gRPC call start_workflow_execution retried 7 times error=Status { code: Unavailable, message: "createOrUpdateCurrentExecution failed. Failed to insert into current_executions table. Error: pq: duplicate key value violates unique constraint \"current_executions_pkey\"", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None }
This suggests that a previous workflow with the same ID was not properly cleaned up, possibly due to a failed rollback transaction in Temporal, resulting in data inconsistency in the current_executions table.
Manual deletion of the row in the database resolves the issue:
DELETE FROM current_executions WHERE workflow_id = 'problem-id';
Steps to Reproduce the Problem
I haven't found a way to reproduce the problem, but the process looks like this:
- Attempt to start a workflow with the given parameters (ALLOW_DUPLICATE, TERMINATE_EXISTING, long timeout).
- Observe retries and eventual failure with the duplicate key value violates unique constraint error.
- Verify that the conflicting workflow ID still exists in current_executions table.
- Manually delete the row to allow the new workflow to start successfully.
Specifications
- Version: temporalio (python sdk) 1.10.0
- Platform: Temporal Version 1.27.1 (PostgreSQL).
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the start_workflow_execution gRPC path and the createOrUpdateCurrentExecution write to PostgreSQL's current_executions table. Trace how TERMINATE_EXISTING and rollback behave across retries, and determine whether the duplicate-key failure can be reproduced or explained as inconsistent state.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, postgresql, python
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100