Race condition between Celery workflow persistence and HITL pause creation
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
### Self Checks
- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.
### Dify version
1.16.1
### Cloud or Self Hosted
Self Hosted (Source)
### Steps to reproduce
1. Configure Dify to use the Celery workflow execution repository:
`CORE_WORKFLOW_EXECUTION_REPOSITORY=core.repositories.celery_workflow_execution_repository.CeleryWorkflowExecutionRepository`
2. Create and publish a workflow containing:
`Start → Human Input → End`
3. Delay consumption of the workflow_storage Celery queue. This can be reproduced by stopping the worker consuming that queue, creating a queue backlog, or adding an artificial delay to save_workflow_execution_task.
4. Run the workflow and let it reach the Human Input node before the asynchronous WorkflowRun save task is processed.
5. Observe the API/worker logs. The pause persistence path fails with an error similar to:
`ValueError: WorkflowRun not found: `
6. Resume consumption of the workflow_storage queue. The WorkflowRun may then be created asynchronously, but the corresponding WorkflowPause and resumption snapshot are still missing.
This issue may also occur intermittently under normal load when the workflow reaches the Human Input node faster than the Celery worker persists the initial WorkflowRun.
### ✔️ Expected Behavior
The workflow should enter the paused state reliably regardless of the configured workflow execution repository.
Before the Human Input pause is exposed to users, Dify should guarantee that:
- The WorkflowRun exists in the database.
- The workflow status is persisted as PAUSED.
- The WorkflowPause, pause reasons, and resumption snapshot are persisted successfully.
- Human Input notifications or forms are made available only after the pause state is durable.
- The workflow can be resumed after a user submits the Human Input form.
Possible valid implementations include synchronously creating the initial WorkflowRun, waiting for asynchronous persistence before creating the pause, or rejecting the Celery repository for pause-capable workflows.
### ❌ Actual Behavior
CeleryWorkflowExecutionRepository.save() enqueues save_workflow_execution_task and returns immediately without waiting for the database transaction to complete.
When the workflow reaches the Human Input node, the following race can occur:
```
GraphRunStarted
→ enqueue async save for WorkflowRun(RUNNING)
GraphRunPaused
→ enqueue async save for WorkflowRun(PAUSED)
PauseStatePersistenceLayer
→ create_workflow_pause()
→ query WorkflowRun directly from the database
→ WorkflowRun does not exist yet
→ ValueError
```
The retries configured on save_workflow_execution_task do not resolve this race because they only retry persistence of the WorkflowRun; they do not retry the failed pause creation.
As a result, the WorkflowRun may eventually appear in the database, but the workflow has no durable pause record or resumption snapshot. The HITL workflow can therefore become inconsistent and may not be resumable.
Contributor guide
Research direction
Start at CeleryWorkflowExecutionRepository.save and save_workflow_execution_task, then trace create_workflow_pause through PauseStatePersistenceLayer. Reproduce with a delayed workflow_storage queue and a Start → Human Input → End workflow. Done means the WorkflowRun and durable pause state exist before the Human Input pause is exposed, and the workflow can be resumed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100