[Usage]: Observing very less throughput-[04/10/2026-11:43:27] [TRT-LLM] [E] Streaming: Failed to process token batch of 10 tokens for request 2

System Info

System Information:

• OS:
• Python version:
• CUDA version:12.9
• GPU model(s): H100
• Driver version:
• TensorRT-LLM version:
  nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10

Detailed output:

I have deployed gpt-oss-120b model using pytorch backend on H100

Please find the deployment here:

spec:
containers:
- args:
- |
cat > /tmp/gpt-oss-120b-h200-tp4-opt.yaml <<'EOF'
env_overrides:
TRTLLM_ENABLE_PDL: 1
enable_chunked_prefill: true
max_num_tokens: 32768
max_batch_size: 256
stream_interval: 10
enable_attention_dp: true
moe_config:
backend: TRITON
num_postprocess_workers: 4
tensor_parallel_size: 4
moe_expert_parallel_size: 4
trust_remote_code: true
max_input_len: 128000
max_seq_len: 128000
kv_cache_config:
free_gpu_memory_fraction: 0.9
EOF

      trtllm-serve openai/gpt-oss-120b --host 0.0.0.0 --port 8000 --tp_size 4 --backend pytorch --config /tmp/gpt-oss-120b-h200-tp4-opt.yaml
    command:
    - bash
    - -lc
    env:
    - name: TRTLLM_ENABLE_PDL
      value: "1"
    - name: HF_HOME
      value: /cache/huggingface
    - name: HUGGINGFACE_HUB_CACHE
      value: /cache/huggingface/hub
    image: nvcr.io/nvidia/tensorrt-llm/release:1.3.0rc10

How would you like to use TensorRT-LLM

I am using trt-llm serve with above deployment config:

I am getting very low throughput:
input/output/concurrency--- 1000/1000/1
The throughput observed was 112.6

As I am new to using TensorRT-LLM with pytorch backend.

Can you please suggest if there is any issue with deployment.

Many thanks in advance.

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.

No repository file, test, or entry point is named; start by reproducing the provided trtllm-serve deployment on H100 with the listed image and configuration. Compare the reported 112.6 throughput with an agreed baseline and collect the missing system details before determining whether a code change is warranted.