Non-leader worker initialization failure can wedge MPI ranks
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
Description
When worker_main fails while constructing GenerationExecutorWorker on a non-leader rank, that rank returns while rank 0 can remain blocked in an MPI collective that the failed rank can no longer enter. The executor process then remains wedged instead of failing promptly, so the serving worker may require manual termination.
Proposed fix
Abort the MPI communicator from a non-leader rank when worker initialization raises. Rank 0 retains the existing behavior of reporting the initialization exception through worker_init_status_queue.
Validation
- Focused unit regression covering the non-leader failure path.
- Real 4-rank MPI integration test with rank 3 constructor failure: rank 3 invoked
MPI_Abort, and the MPI job exited with code 1 before a 20-second timeout.
This was originally observed with disaggregated TensorRT-LLM serving, where a rank initialization failure otherwise left the node-level worker wedged.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating worker_main and the GenerationExecutorWorker construction path, then trace how non-leader ranks participate in the MPI collective and how rank 0 reports through worker_init_status_queue. Add a focused regression for non-leader initialization failure and run the real 4-rank MPI validation. Done means the failed rank aborts the communicator while rank 0 retains its existing reporting behavior and the job exits promptly.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100