[Bug]: V1 MAX_UTILIZATION paused_requests are never processed in the PP executor loop
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
System Info
N/A — identified through source inspection of current main (while working on #17882 / PR #18113).
Reproduction
Code inspection of tensorrt_llm/_torch/pyexecutor/py_executor.py:
- With
pp_size > 1, the executor always selects_executor_loop_pp. _executor_loop_pponly handles the V2 recompute-pause flow (_terminate_recompute_paused_requests/_pause_recompute_paused_requests).- V1
scheduled_batch.paused_requests— produced by the MAX_UTILIZATION capacity scheduler (both the PythonMaxUtilizationPolicyand the C++ bind scheduler, which run withtwo_step_lookaheadunder PP) — are neither passed to_terminate_requestsnor to_pause_requestsanywhere in that loop. The single-rank loops (_executor_loop,_executor_loop_overlap) do both.
Expected behavior
When the V1 MAX_UTILIZATION scheduler decides to pause a started request under pipeline parallelism, the request's resources should be freed and the request reset for replay (req.pause(...)), as in the non-PP loops.
Actual behavior
Under PP, a "paused" request is simply not scheduled that iteration: its KV/resources are never released and its state is never reset, so the preemption the scheduler planned for never materializes. Under sustained memory pressure this can leave the scheduler unable to make room (the same requests get re-paused every iteration while still holding their blocks).
Additional notes
Found while fixing #17882: PR #18113 preserves multimodal payloads for pause replay and threads a for_pause flag through the (PP-)termination path, but the PP loop currently has no V1 pause flow to call it from. Wiring paused_requests handling into _executor_loop_pp touches PP microbatch/inflight-id semantics for all request types, so it is kept out of that PR and tracked here instead.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in tensorrt_llm/_torch/pyexecutor/py_executor.py at _executor_loop_pp, then compare its request handling with _executor_loop and _executor_loop_overlap. Trace scheduled_batch.paused_requests for the V1 MAX_UTILIZATION scheduler and the PP microbatch/inflight-id flow. Done means paused requests release resources and reset for replay under PP without breaking existing request types.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- backend, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100