NVIDIA / NVIDIA/TensorRT-LLM

Non-leader worker initialization failure can wedge MPI ranks

Open
#16,342 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Disaggregated serving
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

Description

When worker_main fails while constructing GenerationExecutorWorker on a non-leader rank, that rank returns while rank 0 can remain blocked in an MPI collective that the failed rank can no longer enter. The executor process then remains wedged instead of failing promptly, so the serving worker may require manual termination.

Proposed fix

Abort the MPI communicator from a non-leader rank when worker initialization raises. Rank 0 retains the existing behavior of reporting the initialization exception through worker_init_status_queue.

Validation

  • Focused unit regression covering the non-leader failure path.
  • Real 4-rank MPI integration test with rank 3 constructor failure: rank 3 invoked MPI_Abort, and the MPI job exited with code 1 before a 20-second timeout.

This was originally observed with disaggregated TensorRT-LLM serving, where a rank initialization failure otherwise left the node-level worker wedged.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating worker_main and the GenerationExecutorWorker construction path, then trace how non-leader ranks participate in the MPI collective and how rank 0 reports through worker_init_status_queue. Add a focused regression for non-leader initialization failure and run the real 4-rank MPI validation. Done means the failed rank aborts the communicator while rank 0 retains its existing reporting behavior and the job exits promptly.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.