conductor-oss / conductor-oss/conductor

PostgresLockDAO does not support recursive locks, causing stuck workflows

Open
#588 10 comments 3 reactions 2 assignees Claimed by @manan164 View on GitHub
bug
Dominant language
Java
Stars
32.2k
Forks
1k
Avg merge
2d 3h
Merged PRs (30d)
37

Description

### PostgresLockDAO does not support recursive locks, causing stuck workflows when sweeper runs with workflow-execution-lock.type=postgres

We’ve encountered a reproducible issue when using **Conductor OSS 3.21.x** with the configuration:

```yaml
conductor.workflow-execution-lock.type=postgres
```

#### Problem

* When all tasks in a workflow are completed, the workflow remains stuck in the `RUNNING` state.
* It never transitions to `COMPLETED` automatically.
* Manually **Pause → Resume** resolves the problem and finalizes the workflow.
* In logs, we repeatedly see messages like:

```
Workflow with id can not be found. Attempting to unack using the id
```

#### Root cause analysis

* The issue is related to **recursive lock acquisition**.
* `WorkflowSweeper` acquires a workflow execution lock [[here](https://github.com/conductor-oss/conductor/blob/main/core/src/main/java/com/netflix/conductor/core/reconciliation/WorkflowSweeper.java#L84-L86)](https://github.com/conductor-oss/conductor/blob/main/core/src/main/java/com/netflix/conductor/core/reconciliation/WorkflowSweeper.java#L84-L86).
* Later, `WorkflowExecutorOps.decide` is called, which tries to acquire the **same lock again** [[here](https://github.com/conductor-oss/conductor/blob/main/core/src/main/java/com/netflix/conductor/core/execution/WorkflowExecutorOps.java#L1051-L1053)](https://github.com/conductor-oss/conductor/blob/main/core/src/main/java/com/netflix/conductor/core/execution/WorkflowExecutorOps.java#L1051-L1053).
* Since `PostgresLockDAO` does **not support recursive locks**, the second attempt fails, causing Sweeper to abandon processing. As a result, the workflow never transitions to `COMPLETED`.

#### Impact

* Workflows stay in `RUNNING` indefinitely even though all tasks are completed.
* Dashboards and monitoring become inaccurate (e.g., “latest workflow status” stuck).
* Requires manual operator intervention (`pause/resume`) to fix.

#### Expected behavior

* Postgres-based locks should either support recursive acquisition or the Conductor code path should be adjusted to avoid double-locking during Sweeper execution.

#### Steps to reproduce

1. Configure Conductor with:

```yaml
conductor.workflow-execution-lock.type=postgres
```
2. Run a workflow with fork/join or subworkflow tasks.
3. Wait until all tasks complete.
4. Observe: workflow stays in `RUNNING` instead of moving to `COMPLETED`.
5. Logs show `Workflow with id can not be found. Attempting to unack using the id`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.