qwen3-8b is running with vLLM's default scheduler limits

What

max_num_batched_tokens defaults to 2048 in vLLM 0.25.1. Every box in the fleet was launched without overriding it, and it throttles admitted concurrency severely — the scheduler will not take on new work once a step's token budget is spent, regardless of how much KV cache is free.

Measured on qwen3.8-27b before and after raising it to 16384:

before after
max concurrent running 2 38
KV cache used 7% 23%
queue time : prefill time 4.3 : 1 —

A 19× improvement in admitted concurrency from one flag. gemma4-26b showed the same pattern — Running: 1 with up to 46 queued while its cache sat at 5%.

Both have since been relaunched with --enable-chunked-prefill --max-num-batched-tokens 16384. qwen3-8b has not.

Why it matters more here, not less

The 8B is the tier meant to absorb routine, high-volume traffic. It is also on a 24GB card, so it has the least cache to admit into — which makes efficient scheduling more important, not less.

Fix

Recreate the pod with the two flags added:

--enable-chunked-prefill --max-num-batched-tokens 16384

--max-num-partial-prefills is not supported in this version — it raises NotImplementedError: Concurrent Partial Prefill. Do not include it.

The catalog entry needs the same flags in model_info.vllm.args, or a reprovision reverts to the defaults.

Worth measuring, not assuming

16384 was chosen to make the 27B's problem go away, not because it is optimal. The 8B has a 40,960 window and a smaller cache, so a different value may suit it. Worth ramping before and after rather than copying the number across.

The tradeoff is decode latency for requests already running: larger prefill batches mean an in-flight token waits longer behind new arrivals. At a 4.3:1 queue-to-prefill ratio that was obviously worth it; on a less loaded box it may not be.

Note

Every throughput figure gathered before this fix was measured through the bottleneck and should be re-taken.

Start with the qwen3-8b pod configuration and its catalog entry at model_info.vllm.args; compare them with the already relaunched 27B and gemma4-26b deployments. Reprovision with chunked prefill and a measured max_num_batched_tokens value, excluding max-num-partial-prefills, then re-run throughput, queue, prefill, concurrency, and latency measurements.

Written by the indexing model from the issue text.