[Feature]: PyExecutor multi-stage pipelined scheduler (matching vLLM v1's step_with_batch_queue)

🚀 The feature, motivation and pitch

PyExecutor._executor_loop_overlap runs the scheduler, forward dispatch, sampler post-processing, and request management on a single Python thread with at most 1-step lookahead via self.previous_batch. On steady-state decode workloads where per-iter host work is comparable to GPU work, this exposes ~580 µs/iter of host overhead as GPU idle, capping aggregate throughput. Replace this with a vLLM-v1-style pipelined scheduler that maintains a 2-batch in-flight queue served by a worker thread, so the scheduler is structurally one batch ahead and host work is hidden behind GPU work.

Workload: Llama-3.1-8B-Instruct-FP8, B200, ISL=OSL=1000, conc=64, OTPS bench.
TRT-LLM AD 14,069 OTPS
TRT-LLM PyTorch backend 13,806 OTPS
vLLM 0.20.1 15,638 OTPS

traces:
/home/scratch.egeva_coreai/perf_analysis/no_mintok_260507/ad_traces/ad_phk/trace_ad_ad_phk.nsys-rep
/home/scratch.egeva_coreai/perf_analysis/no_mintok_260507/vllm_traces/vl_gap/trace_vllm_vl_gap.nsys-rep

Alternatives

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.