NVIDIA / NVIDIA/TensorRT-LLM

[Bug]: V1 MAX_UTILIZATION paused_requests are never processed in the PP executor loop

Open
#18,115 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Inference runtime
Dominant language
Python
Stars
14.7k
Forks
2.8k
Avg merge
2d 23h
Merged PRs (30d)
489

Description

System Info

N/A — identified through source inspection of current main (while working on #17882 / PR #18113).

Reproduction

Code inspection of tensorrt_llm/_torch/pyexecutor/py_executor.py:

  • With pp_size > 1, the executor always selects _executor_loop_pp.
  • _executor_loop_pp only handles the V2 recompute-pause flow (_terminate_recompute_paused_requests / _pause_recompute_paused_requests).
  • V1 scheduled_batch.paused_requests — produced by the MAX_UTILIZATION capacity scheduler (both the Python MaxUtilizationPolicy and the C++ bind scheduler, which run with two_step_lookahead under PP) — are neither passed to _terminate_requests nor to _pause_requests anywhere in that loop. The single-rank loops (_executor_loop, _executor_loop_overlap) do both.
Expected behavior

When the V1 MAX_UTILIZATION scheduler decides to pause a started request under pipeline parallelism, the request's resources should be freed and the request reset for replay (req.pause(...)), as in the non-PP loops.

Actual behavior

Under PP, a "paused" request is simply not scheduled that iteration: its KV/resources are never released and its state is never reset, so the preemption the scheduler planned for never materializes. Under sustained memory pressure this can leave the scheduler unable to make room (the same requests get re-paused every iteration while still holding their blocks).

Additional notes

Found while fixing #17882: PR #18113 preserves multimodal payloads for pause replay and threads a for_pause flag through the (PP-)termination path, but the PP loop currently has no V1 pause flow to call it from. Wiring paused_requests handling into _executor_loop_pp touches PP microbatch/inflight-id semantics for all request types, so it is kept out of that PR and tracked here instead.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in tensorrt_llm/_torch/pyexecutor/py_executor.py at _executor_loop_pp, then compare its request handling with _executor_loop and _executor_loop_overlap. Trace scheduled_batch.paused_requests for the V1 MAX_UTILIZATION scheduler and the PP microbatch/inflight-id flow. Done means paused requests release resources and reset for replay under PP without breaking existing request types.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, python
Domain
backend, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.