[Bug] Attention-DP fill-gate fail-fast can skip the model-parallel status gather on a post-fill rank
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
Description
PyExecutor._sync_gen_only_benchmark_has_insufficient_kv() gates the model-parallel
status gather on rank-local dynamic state:
if (self.benchmark_req_queues_size <= 0 or self.is_warmup
or not self._benchmark_fill_phase_active):
return False
_benchmark_fill_phase_active is seeded from the static benchmark-disagg config but is
cleared per rank when the global fill gate opens (_check_benchmark_disagg_gate,
py_executor.py:3995). A rank that observes the flag clear returns before
_allgather_model_parallel_status(), while a peer that is still filling enters the
gather. The two ranks then run different collectives on the same communicator: the fill
gate gathers a pickled object (MPIDist.tp_allgather) while _can_queue() gathers a raw
MPI.INT64_T (tp_allgather_int64). MPI matches them by order and datatype rather than
failing, so the payloads are silently mismatched — the same failure class already
documented from a live run at py_executor.py:4676-4679.
This is the only site in the tree that gates a collective on rank-local state before
entering it. Its sibling _is_benchmark_disagg_fill_complete() states the required
discipline explicitly (py_executor.py:3903-3905):
Keep going to the shared allgather even when this rank is not done so
model-parallel ranks cannot diverge in collective order.
Steps to reproduce
Two real MPI ranks, with the production PyExecutor method bound to a stub whose dist
is a real MPIDist. Rank 0 keeps _benchmark_fill_phase_active = True; rank 1 clears it.
stub._sync_gen_only_benchmark_has_insufficient_kv([], False)
mpirun -n 2 --oversubscribe python native_vote_probe.py
| rank | unpatched (f9e3e06ee7) |
|---|---|
rank 0 (fill_active=true) |
never returns — blocked inside MPI_Allgather |
rank 1 (fill_active=false) |
returns False in 0.000 s, never entered the gather |
mpirun |
killed by a 90 s outer timeout, rc=124 |
With the flag voted instead of read, both ranks return and agree (mpirun rc=0;
rank 1 spends 1.577 s waiting at the gather, i.e. it actually participates).
Suggested fix direction
Keep only the rank-uniform benchmark_req_queues_size / is_warmup short-circuit,
carry _benchmark_fill_phase_active in the status tuple, and take the fail-fast decision
from the gathered result. Normal serving (TLLM_BENCHMARK_REQ_QUEUES_SIZE unset) still
returns before any collective, so no steady-state synchronisation is added.
Notes
Related to #17095: this is the fill-gate allgather reported there as the first wait
point. Whether it is the whole original deadlock is not established — on this hardware
(8x L20, sm_89) the disaggregated gen-only run stalls before the GEN fill gate is ever
reached, so the original HangDetector -> MPI_Abort signature was not reproduced.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in py_executor.py at _sync_gen_only_benchmark_has_insufficient_kv(), _check_benchmark_disagg_gate(), and _allgather_model_parallel_status(). Run native_vote_probe.py with mpirun -n 2 using a real MPIDist, then verify both ranks enter the shared gather and agree on the fail-fast result. The normal serving path should still return before any collective.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100