opensearch-project / opensearch-project/flow-framework
[BUG]RegisterModelGroupStep can hang indefinitely if state index async update never completes
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 62
- Forks
- 66
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 20
Description
What is the bug?
RegisterModelGroupStep can hang until the node timeout expires (minimum 10 seconds by default) if the async state index update inside FlowFrameworkIndicesHandler.addResourceToStateIndex never completes. In this case, the PlainActionFuture associated with the workflow step is never resolved, causing the workflow node to wait until timeout and then fail with a TimeoutException.
How can one reproduce the bug?
Trigger a workflow that executes RegisterModelGroupStep.
Let mlClient.registerModelGroup complete successfully.
During the subsequent call to FlowFrameworkIndicesHandler.addResourceToStateIndex, simulate a scenario where:
sdkClient.getDataObjectAsync() or sdkClient.updateDataObjectAsync() never completes (e.g. network partition, hung connection, missing client-level timeout).
Observe that the CompletableFuture.whenComplete() callback is never invoked.
The PlainActionFuture in RegisterModelGroupStep is never completed.
What is the expected behavior?
The workflow step should not hang indefinitely if the async state index update stalls.
Instead, the step should fail deterministically after a timeout and allow the workflow engine to handle the failure (retry, fail, or exit based on configuration).
What is your host/environment?
Not environment-specific.
This issue can occur in any environment where async SDK calls do not have an enforced timeout (e.g. network instability or misconfigured client timeouts).
Do you have any screenshots?
N/A
Do you have any additional context?
While ProcessNode.execute() does apply a timeout (default 10 seconds from NODE_TIMEOUT_DEFAULT_VALUE, or the step-specific timeout from WorkflowStepFactory enum), this timeout only protects at the ProcessNode level. If addResourceToStateIndex hangs, the step's PlainActionFuture will not complete, causing the ProcessNode to wait until its timeout expires before failing. This means the workflow will be stuck in RUNNING state for the duration of the timeout period, which could be problematic for long-running workflows or if the timeout is configured to be very long.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace RegisterModelGroupStep through FlowFrameworkIndicesHandler.addResourceToStateIndex and ProcessNode.execute(), starting with the PlainActionFuture completion path and NODE_TIMEOUT_DEFAULT_VALUE. Done means a stalled async SDK update causes the step to fail through the workflow timeout path rather than remain unresolved, with the relevant workflow-step behavior verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- backend, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100