temporalio / temporalio/temporal

start_workflow_execution fails with duplicate key error despite TERMINATE_EXISTING policy (possible DB inconsistency)

Open
#7,600 6 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

potential-bug
Dominant language
Go
Stars
23.2k
Forks
1.9k
Avg merge
2d 8h
Merged PRs (30d)
228

Description

Expected Behavior

Starting a workflow with the following parameters should either succeed or terminate the existing one as specified by the conflict policy:

  • id_reuse_policy: ALLOW_DUPLICATE
  • execution_timeout: 12h
  • id_conflict_policy: TERMINATE_EXISTING

Actual Behavior

The gRPC call start_workflow_execution fails after multiple retries with the following error:

WARN temporal_client::retry: gRPC call start_workflow_execution retried 7 times error=Status { code: Unavailable, message: "createOrUpdateCurrentExecution failed. Failed to insert into current_executions table. Error: pq: duplicate key value violates unique constraint \"current_executions_pkey\"", metadata: MetadataMap { headers: {"content-type": "application/grpc"} }, source: None }

This suggests that a previous workflow with the same ID was not properly cleaned up, possibly due to a failed rollback transaction in Temporal, resulting in data inconsistency in the current_executions table.

Manual deletion of the row in the database resolves the issue:

DELETE FROM current_executions WHERE workflow_id = 'problem-id';

Steps to Reproduce the Problem

I haven't found a way to reproduce the problem, but the process looks like this:

  1. Attempt to start a workflow with the given parameters (ALLOW_DUPLICATE, TERMINATE_EXISTING, long timeout).
  2. Observe retries and eventual failure with the duplicate key value violates unique constraint error.
  3. Verify that the conflicting workflow ID still exists in current_executions table.
  4. Manually delete the row to allow the new workflow to start successfully.

Specifications

  • Version: temporalio (python sdk) 1.10.0
  • Platform: Temporal Version 1.27.1 (PostgreSQL).

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the start_workflow_execution gRPC path and the createOrUpdateCurrentExecution write to PostgreSQL's current_executions table. Trace how TERMINATE_EXISTING and rollback behave across retries, and determine whether the duplicate-key failure can be reproduced or explained as inconsistent state.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, postgresql, python
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.