conductor-oss / conductor-oss/conductor
PostgresLockDAO does not support recursive locks, causing stuck workflows
- Dominant language
- Java
- Stars
- 32.2k
- Forks
- 1k
- Avg merge
- 2d 3h
- Merged PRs (30d)
- 37
Description
### PostgresLockDAO does not support recursive locks, causing stuck workflows when sweeper runs with workflow-execution-lock.type=postgres
We’ve encountered a reproducible issue when using **Conductor OSS 3.21.x** with the configuration:
```yaml
conductor.workflow-execution-lock.type=postgres
```
#### Problem
* When all tasks in a workflow are completed, the workflow remains stuck in the `RUNNING` state.
* It never transitions to `COMPLETED` automatically.
* Manually **Pause → Resume** resolves the problem and finalizes the workflow.
* In logs, we repeatedly see messages like:
```
Workflow with id can not be found. Attempting to unack using the id
```
#### Root cause analysis
* The issue is related to **recursive lock acquisition**.
* `WorkflowSweeper` acquires a workflow execution lock [[here](https://github.com/conductor-oss/conductor/blob/main/core/src/main/java/com/netflix/conductor/core/reconciliation/WorkflowSweeper.java#L84-L86)](https://github.com/conductor-oss/conductor/blob/main/core/src/main/java/com/netflix/conductor/core/reconciliation/WorkflowSweeper.java#L84-L86).
* Later, `WorkflowExecutorOps.decide` is called, which tries to acquire the **same lock again** [[here](https://github.com/conductor-oss/conductor/blob/main/core/src/main/java/com/netflix/conductor/core/execution/WorkflowExecutorOps.java#L1051-L1053)](https://github.com/conductor-oss/conductor/blob/main/core/src/main/java/com/netflix/conductor/core/execution/WorkflowExecutorOps.java#L1051-L1053).
* Since `PostgresLockDAO` does **not support recursive locks**, the second attempt fails, causing Sweeper to abandon processing. As a result, the workflow never transitions to `COMPLETED`.
#### Impact
* Workflows stay in `RUNNING` indefinitely even though all tasks are completed.
* Dashboards and monitoring become inaccurate (e.g., “latest workflow status” stuck).
* Requires manual operator intervention (`pause/resume`) to fix.
#### Expected behavior
* Postgres-based locks should either support recursive acquisition or the Conductor code path should be adjusted to avoid double-locking during Sweeper execution.
#### Steps to reproduce
1. Configure Conductor with:
```yaml
conductor.workflow-execution-lock.type=postgres
```
2. Run a workflow with fork/join or subworkflow tasks.
3. Wait until all tasks complete.
4. Observe: workflow stays in `RUNNING` instead of moving to `COMPLETED`.
5. Logs show `Workflow with id can not be found. Attempting to unack using the id`.
Contributor guide
Assessment
This issue has not been assessed yet.