Parallel Human Input submissions can leave the workflow paused after both forms are submitted
- Dominant language
- TypeScript
- Stars
- 156k
- Forks
- 24.6k
- Avg merge
- 22h 9m
- Merged PRs (30d)
- 610
Description
**AI disclosure**: This issue was drafted and analyzed with Codex. I have reviewed the analysis, and I am responsible for the content.
### Self Checks
- [x] I have read the [Contributing Guide](https://github.com/langgenius/dify/blob/main/CONTRIBUTING.md) and [Language Policy](https://github.com/langgenius/dify/issues/1542).
- [x] This is only for bug report, if you would like to ask a question, please head to [Discussions](https://github.com/langgenius/dify/discussions/categories/general).
- [x] I have searched for existing issues [search for existing issues](https://github.com/langgenius/dify/issues), including closed ones.
- [x] I confirm that I am using English to submit this report, otherwise it will be closed.
- [x] 【中文用户 & Non English User】请使用英语提交,否则会被关闭 :)
- [x] Please do not modify this template :) and fill in all the required fields.
### Dify version
1.17.0, self-hosted source checkout, Graphon 0.7.0.
Exact tested checkout: [6ec7dc60b97c7b025c21ad91c634b4c52abfd45d](https://github.com/QuantumGhost/dify/commit/6ec7dc60b97c7b025c21ad91c634b4c52abfd45d), including the HITL completion-event fix in #42082. This report does **not** claim verification on an unmodified 1.17.0 release or current upstream HEAD.
### Cloud or Self Hosted
Self Hosted (Source)
### Steps to reproduce
Two parallel Human Input forms can both be accepted by the server, but overlapping resume tasks can leave the workflow paused with no further effective resume.
1. Use a Celery worker with at least two concurrent execution slots. The reproduction used gevent with `-c 2`, consuming `workflow_based_app_execution,workflow,workflow_storage,conversation`, with real PostgreSQL and Redis.
2. Save the complete DSL below as `parallel-hitl-resume-race.yml` and import it into Dify. It contains four nodes and requires no model/provider configuration:
```text
Start -> HITL 1 -> Answer
Start -> HITL 2 -> Answer
```
Both branches join the **same** Answer node. Its template references HITL 1 before HITL 2.
3. Open Studio Preview, or publish the app and open the Published WebApp. Send a message and wait until both **Approve 1** and **Approve 2** are visible.
4. Click **Approve 1**, then **Approve 2** immediately, without waiting for the first resume to complete and pause again. Confirm in the network panel that both form submission POST requests were sent and accepted. Repeat with a fresh run if necessary: this is timing-dependent.
5. Check the Celery logs, workflow status, and both form records. On an affected run, both forms are `submitted`, but the workflow remains `paused`, the second form stays visible, and the answer stops before the second action value.
The browser reproduction used sequential Playwright clicks with no deliberate wait between them:
```javascript
await page.getByRole('button', { name: 'Approve 1', exact: true }).click();
await page.getByRole('button', { name: 'Approve 2', exact: true }).click();
```
For comparison, submitting one form, waiting for the resume/re-pause, and then submitting the other succeeded in **both orders** (`1 → 2` and `2 → 1`) in Studio and Published WebApp. Final output was always:
```text
Joined: | H1=approve1 | H2=approve2
```
The tests used real Chrome, frontend, API, PostgreSQL, Redis, and Celery. Requests and SSE were not mocked.
Minimal reproducible DSL — save as parallel-hitl-resume-race.yml
```yaml
app:
description: Local regression verification for HITL completion events.
icon: H
icon_type: emoji
icon_background: '#E0EAFF'
mode: advanced-chat
name: HITL Parallel
use_icon_as_answer_icon: false
dependencies: []
kind: app
version: 0.7.0
workflow:
conversation_variables: []
environment_variables: []
features:
file_upload:
enabled: false
opening_statement: ''
retriever_resource:
enabled: false
sensitive_word_avoidance:
enabled: false
speech_to_text:
enabled: false
suggested_questions: []
suggested_questions_after_answer:
enabled: false
text_to_speech:
enabled: false
graph:
nodes:
- id: start
type: custom
data:
type: start
desc: ''
selected: false
title: Start
variables: []
position:
x: 80
y: 250
positionAbsolute:
x: 80
y: 250
width: 244
height: 120
sourcePosition: right
targetPosition: left
- id: human1
type: custom
data:
type: human-input
desc: ''
selected: false
title: HITL 1
form_content: 'HITL 1: {{#sys.query#}}'
inputs: []
user_actions:
- id: approve1
title: Approve 1
button_style: primary
delivery_methods:
- id: ce75d9ad-007a-4fd1-9e7a-b67bfa69f4ce
type: webapp
enabled: true
config: {}
timeout: 3
timeout_unit: day
position:
x: 440
y: 250
positionAbsolute:
x: 440
y: 250
width: 244
height: 120
sourcePosition: right
targetPosition: left
- id: human2
type: custom
data:
type: human-input
desc: ''
selected: false
title: HITL 2
form_content: 'HITL 2: {{#sys.query#}}'
inputs: []
user_actions:
- id: approve2
title: Approve 2
button_style: primary
delivery_methods:
- id: 383a6d76-0040-4c98-8b27-4d584bb7241d
type: webapp
enabled: true
config: {}
timeout: 3
timeout_unit: day
position:
x: 800
y: 250
positionAbsolute:
x: 800
y: 250
width: 244
height: 120
sourcePosition: right
targetPosition: left
- id: answer
type: custom
data:
type: answer
desc: ''
selected: false
title: Answer
answer: 'Joined: {{#sys.query#}} | H1={{#human1.__action_id#}} | H2={{#human2.__action_id#}}'
variables: []
position:
x: 1160
y: 250
positionAbsolute:
x: 1160
y: 250
width: 244
height: 120
sourcePosition: right
targetPosition: left
edges:
- id: start-source-human1
source: start
sourceHandle: source
target: human1
targetHandle: target
type: custom
data:
isInIteration: false
isInLoop: false
sourceType: start
targetType: human-input
zIndex: 0
- id: start-source-human2
source: start
sourceHandle: source
target: human2
targetHandle: target
type: custom
data:
isInIteration: false
isInLoop: false
sourceType: start
targetType: human-input
zIndex: 0
- id: human1-approve1-answer
source: human1
sourceHandle: approve1
target: answer
targetHandle: target
type: custom
data:
isInIteration: false
isInLoop: false
sourceType: human-input
targetType: answer
zIndex: 0
- id: human2-approve2-answer
source: human2
sourceHandle: approve2
target: answer
targetHandle: target
type: custom
data:
isInIteration: false
isInLoop: false
sourceType: human-input
targetType: answer
zIndex: 0
viewport:
x: 0
y: 0
zoom: 0.8
```
### ✔️ Expected Behavior
After both form submissions have been accepted, the workflow should eventually process both submissions, execute the shared Answer node once, and finish successfully. Concurrent resume requests for the same workflow should not lose the second submission's effective wake-up.
The result should be the same whether the forms are submitted slowly, in reverse order, or in rapid succession. Once complete, both forms should be shown as submitted and the WebApp should accept the next message.
### ❌ Actual Behavior
A rapid-submission run in Studio ended in this state, confirmed in PostgreSQL after the worker tasks had returned:
| Workflow run | Workflow status | Form | Form status | Selected action |
| --- | --- | --- | --- | --- |
| `32baf543-819b-433b-a103-bcdf406127f1` | `paused` | `human1` | `submitted` | `approve1` |
| `32baf543-819b-433b-a103-bcdf406127f1` | `paused` | `human2` | `submitted` | `approve2` |
The persisted answer was incomplete:
```text
Joined: unbuffered-studio-1789009924109-12-rapid | H1=approve1 | H2=
```
The remaining approval button did not disappear. This is more than a missing frontend notification: the database still showed a paused workflow even though both form submissions were already persisted.
**Observed worker sequence** (relevant lines; local filesystem prefixes omitted):
```text
03:12:04.715 Task resume_app_execution[265f2a8b-0e10-44e6-babf-43f524e4702b] received
03:12:04.726 Resumed workflow pause for workflow run 32baf543-819b-433b-a103-bcdf406127f1
03:12:04.749 Task resume_app_execution[bf0d6dae-bf7f-4cc2-a98c-bd22aab5451a] received
03:12:04.776 _WorkflowRunError: WorkflowRun is not in PAUSED status,
workflow_run_id=32baf543-819b-433b-a103-bcdf406127f1, current_status=running
03:12:04.898 Created another workflow pause for workflow run 32baf543-819b-433b-a103-bcdf406127f1
03:12:04.941 Task resume_app_execution[265f2a8b-0e10-44e6-babf-43f524e4702b] succeeded
```
Relevant traceback:
```text
api/tasks/app_generate/workflow_execute_task.py:760, in resume_app_execution
_resume_app_execution(payload)
api/tasks/app_generate/workflow_execute_task.py:568, in _resume_app_execution
workflow_run_repo.resume_workflow_pause(workflow_run_id, pause_entity)
api/repositories/sqlalchemy_api_workflow_run_repository.py:1188, in resume_workflow_pause
raise _WorkflowRunError(...)
```
The first attempt then creates another pause. The second form's accepted submission has no effective resume left to process it. In another rapid-submission run, a duplicate resume failed with the same error but the other attempt happened to see both submissions and completed; therefore a duplicate-task error alone does not always imply the workflow is stuck.
**Likely cause, based on the observed sequence and code:**
- [`HumanInputService.submit_form_by_token`](https://github.com/QuantumGhost/dify/blob/6ec7dc60b97c7b025c21ad91c634b4c52abfd45d/api/services/human_input_service.py#L215) persists each form submission and independently enqueues a resume for the same workflow run.
- [`_resume_app_execution`](https://github.com/QuantumGhost/dify/blob/6ec7dc60b97c7b025c21ad91c634b4c52abfd45d/api/tasks/app_generate/workflow_execute_task.py#L568) calls `resume_workflow_pause` before running the resumed graph.
- [`resume_workflow_pause`](https://github.com/QuantumGhost/dify/blob/6ec7dc60b97c7b025c21ad91c634b4c52abfd45d/api/repositories/sqlalchemy_api_workflow_run_repository.py#L1187) rejects the second attempt when the first has already changed the workflow to `running`. The observed failing task does not supply a later wake-up after the first attempt pauses again.
**Control comparison / relation to other issues:**
This was found while verifying #42082, but restoring its previous buffered `HumanInputFormEventFilter` from `e2a3698c93e` in otherwise identical test processes also reproduced the stuck state in Published WebApp: run `f5035cdb-54d7-41e9-8ee9-1e184e4f725f` was `paused` with both forms `submitted`. Its overlapping resume failed with the same `current_status=running` error. Thus restoring traversal buffering does not prevent this race.
This differs from #42062, where a workflow can succeed while the form notification is missing, and from #40445, which concerns initial workflow/pause persistence. Here the initial pause exists and the failure is during overlapping resumes after two accepted submissions.
Contributor guide
Research direction
Reproduce the race with the supplied parallel HITL DSL and a Celery worker with concurrent slots. Read HumanInputService.submit_form_by_token, _resume_app_execution in api/tasks/app_generate/workflow_execute_task.py, and resume_workflow_pause in api/repositories/sqlalchemy_api_workflow_run_repository.py. Done means rapid submissions process both forms, execute the shared Answer once, finish the workflow, and accept the next message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- postgresql, python, redis
- Domain
- api, backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100