deepmodeling / deepmodeling/dpdispatcher

SSH cleanup race: rm -rf Directory not empty aborts ratio_unfinished submission

Closed
#682 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
62
Forks
63
Avg merge
2d 22h
Merged PRs (30d)
36

Description

## Bug: SSH cleanup fails with `Directory not empty` after `ratio_unfinished` early termination

### Summary
With an SSH context on a NAS-backed remote filesystem, a submission using `strategy.ratio_unfinished > 0` can reach the unfinished-task threshold, call `remove_unfinished_tasks()`, and then fail in `SSHContext._rmtree()` during `rm -rf`. The non-zero return raises `RuntimeError` and aborts the DP-GEN stage even though completed task results were already downloaded.

### Observed behavior
- 32 model-deviation tasks; 26 finished (81.25%).
- `ratio_unfinished = 0.2` triggered `remove_unfinished_tasks()`.
- Remaining task directories were still being finalized on the remote NAS.
- `rm -rf ` returned: `rm: cannot remove .../task.000.000000: Directory not empty`.
- `SSHContext._rmtree()` propagated the error; the parent DP-GEN process exited with `RuntimeError`.
- 26 completed `model_devi.out` files had already been downloaded.

### Environment
- dpdispatcher 1.0.3
- SSH context
- remote root on NAS-backed storage
- Python 3.12

### Expected behavior
Cleanup should tolerate transient `ENOTEMPTY`/directory metadata races (for example, retry with backoff or use best-effort asynchronous cleanup) and return the completed submission state instead of aborting the workflow.

### Relevant code
- `dpdispatcher/submission.py::remove_unfinished_tasks`
- `dpdispatcher/contexts/ssh_context.py::_rmtree`

The issue is reproducible when unfinished tasks are still writing restart/finalization files as cleanup starts.

Contributor guide

Open the contributing guide

Research direction

Start with dpdispatcher/contexts/ssh_context.py::_rmtree and dpdispatcher/submission.py::remove_unfinished_tasks; inspect how the remote rm -rf result becomes RuntimeError. Reproduce or test the ratio_unfinished cleanup path while unfinished remote tasks are still writing, then verify transient “Directory not empty” failures are tolerated and completed submission state returns without aborting.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
hpc, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.