deepmodeling / deepmodeling/dpdispatcher
SSH cleanup race: rm -rf Directory not empty aborts ratio_unfinished submission
- Dominant language
- Python
- Stars
- 62
- Forks
- 63
- Avg merge
- 2d 22h
- Merged PRs (30d)
- 36
Description
## Bug: SSH cleanup fails with `Directory not empty` after `ratio_unfinished` early termination
### Summary
With an SSH context on a NAS-backed remote filesystem, a submission using `strategy.ratio_unfinished > 0` can reach the unfinished-task threshold, call `remove_unfinished_tasks()`, and then fail in `SSHContext._rmtree()` during `rm -rf`. The non-zero return raises `RuntimeError` and aborts the DP-GEN stage even though completed task results were already downloaded.
### Observed behavior
- 32 model-deviation tasks; 26 finished (81.25%).
- `ratio_unfinished = 0.2` triggered `remove_unfinished_tasks()`.
- Remaining task directories were still being finalized on the remote NAS.
- `rm -rf ` returned: `rm: cannot remove .../task.000.000000: Directory not empty`.
- `SSHContext._rmtree()` propagated the error; the parent DP-GEN process exited with `RuntimeError`.
- 26 completed `model_devi.out` files had already been downloaded.
### Environment
- dpdispatcher 1.0.3
- SSH context
- remote root on NAS-backed storage
- Python 3.12
### Expected behavior
Cleanup should tolerate transient `ENOTEMPTY`/directory metadata races (for example, retry with backoff or use best-effort asynchronous cleanup) and return the completed submission state instead of aborting the workflow.
### Relevant code
- `dpdispatcher/submission.py::remove_unfinished_tasks`
- `dpdispatcher/contexts/ssh_context.py::_rmtree`
The issue is reproducible when unfinished tasks are still writing restart/finalization files as cleanup starts.
Contributor guide
Research direction
Start with dpdispatcher/contexts/ssh_context.py::_rmtree and dpdispatcher/submission.py::remove_unfinished_tasks; inspect how the remote rm -rf result becomes RuntimeError. Reproduce or test the ratio_unfinished cleanup path while unfinished remote tasks are still writing, then verify transient “Directory not empty” failures are tolerated and completed submission state returns without aborting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- hpc, infrastructure
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100