NVIDIA / NVIDIA/TensorRT-LLM

[Bug] Attention-DP fill-gate fail-fast can skip the model-parallel status gather on a post-fill rank

Open
#19,435 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Disaggregated serving
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

Description

PyExecutor._sync_gen_only_benchmark_has_insufficient_kv() gates the model-parallel
status gather on rank-local dynamic state:

if (self.benchmark_req_queues_size <= 0 or self.is_warmup
        or not self._benchmark_fill_phase_active):
    return False

_benchmark_fill_phase_active is seeded from the static benchmark-disagg config but is
cleared per rank when the global fill gate opens (_check_benchmark_disagg_gate,
py_executor.py:3995). A rank that observes the flag clear returns before
_allgather_model_parallel_status(), while a peer that is still filling enters the
gather. The two ranks then run different collectives on the same communicator: the fill
gate gathers a pickled object (MPIDist.tp_allgather) while _can_queue() gathers a raw
MPI.INT64_T (tp_allgather_int64). MPI matches them by order and datatype rather than
failing, so the payloads are silently mismatched — the same failure class already
documented from a live run at py_executor.py:4676-4679.

This is the only site in the tree that gates a collective on rank-local state before
entering it. Its sibling _is_benchmark_disagg_fill_complete() states the required
discipline explicitly (py_executor.py:3903-3905):

Keep going to the shared allgather even when this rank is not done so
model-parallel ranks cannot diverge in collective order.

Steps to reproduce

Two real MPI ranks, with the production PyExecutor method bound to a stub whose dist
is a real MPIDist. Rank 0 keeps _benchmark_fill_phase_active = True; rank 1 clears it.

stub._sync_gen_only_benchmark_has_insufficient_kv([], False)
mpirun -n 2 --oversubscribe python native_vote_probe.py
rank unpatched (f9e3e06ee7)
rank 0 (fill_active=true) never returns — blocked inside MPI_Allgather
rank 1 (fill_active=false) returns False in 0.000 s, never entered the gather
mpirun killed by a 90 s outer timeout, rc=124

With the flag voted instead of read, both ranks return and agree (mpirun rc=0;
rank 1 spends 1.577 s waiting at the gather, i.e. it actually participates).

Suggested fix direction

Keep only the rank-uniform benchmark_req_queues_size / is_warmup short-circuit,
carry _benchmark_fill_phase_active in the status tuple, and take the fail-fast decision
from the gathered result. Normal serving (TLLM_BENCHMARK_REQ_QUEUES_SIZE unset) still
returns before any collective, so no steady-state synchronisation is added.

Notes

Related to #17095: this is the fill-gate allgather reported there as the first wait
point. Whether it is the whole original deadlock is not established — on this hardware
(8x L20, sm_89) the disaggregated gen-only run stalls before the GEN fill gate is ever
reached, so the original HangDetector -> MPI_Abort signature was not reproduced.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in py_executor.py at _sync_gen_only_benchmark_has_insufficient_kv(), _check_benchmark_disagg_gate(), and _allgather_model_parallel_status(). Run native_vote_probe.py with mpirun -n 2 using a real MPIDist, then verify both ranks enter the shared gather and agree on the fail-fast result. The normal serving path should still return before any collective.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.