With 8-H100, trtllm couldn't host qwen3 moe 235B

System Info

Trtllm v0.20.0

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Trtllm version: v0.20.0,
Expectation: From this blog: https://developer.nvidia.com/zh-cn/blog/modelscope-nvidia-tensorrt-llm-pytorch-qwen3/ , we can check that trtllm claimed support of qwen3 moe 235B on 8 GPUs. But I got OOM.
Demo script:

from tensorrt_llm._torch import LLM
from tensorrt_llm import SamplingParams


def main():

    # Model could accept HF model name, a path to local HF model,
    # or TensorRT Model Optimizer's quantized checkpoints like nvidia/Llama-3.1-8B-Instruct-FP8 on HF.
    # model_path = "Qwen/Qwen3-30B-A3B"
    model_path = "Qwen/Qwen3-235B-A22B"
    tp = 8
    llm = LLM(model=model_path, tensor_parallel_size=tp, moe_tensor_parallel_size=1, moe_expert_parallel_size=tp, max_num_tokens=1160, max_batch_size=161)

    # Sample prompts.
    prompts = [
        "Hello, my name is",
    ]

    # Create a sampling params.
    sampling_params = SamplingParams(temperature=0.8, top_p=0.95, n = 1)

    for output in llm.generate(prompts, sampling_params):
        print(
            f"Prompt: {output.prompt!r}, Generated text: {output.outputs[0].text!r}"
        )

if __name__ == '__main__':
    main()

Error:

[07/28/2025-07:59:02] [TRT-LLM] [E] Traceback (most recent call last):
  File "/root/nvda/TensorRT-LLM/tensorrt_llm/executor/worker.py", line 698, in worker_main
    worker: ExecutorBindingsWorker = worker_cls(
  File "/root/nvda/TensorRT-LLM/tensorrt_llm/executor/worker.py", line 128, in __init__
    self.engine = _create_engine()
  File "/root/nvda/TensorRT-LLM/tensorrt_llm/executor/worker.py", line 126, in _create_engine
    return create_executor(**args)
  File "/root/nvda/TensorRT-LLM/tensorrt_llm/_torch/pyexecutor/py_executor_creator.py", line 207, in create_py_executor
    kv_cache_manager = create_kv_cache_manager(model_engine, mapping,
  File "/root/nvda/TensorRT-LLM/tensorrt_llm/_torch/pyexecutor/_util.py", line 301, in create_kv_cache_manager
    kv_cache_manager = KVCacheManager(
  File "/root/nvda/TensorRT-LLM/tensorrt_llm/_torch/pyexecutor/resource_manager.py", line 231, in __init__
    self.impl.allocate_pools(False)
RuntimeError: [TensorRT-LLM][ERROR] CUDA runtime error in ::cudaMalloc(ptr, n): out of memory (/home/jenkins/agent/workspace/LLM/release-0.20/L0_Test-x86_64/tensorrt_llm/cpp/tensorrt_llm/runtime/tllmBuffers.h:92)

Expected behavior

The model shoul be successfully hosted. Because 8 H100 is enough for this model.

actual behavior

OOM reported by trtllm.

additional notes

No more.

Start by reproducing the Qwen/Qwen3-235B-A22B configuration from the issue with TensorRT-LLM v0.20.0 and 8 H100 GPUs. Trace KV-cache allocation through tensorrt_llm/_torch/pyexecutor/py_executor_creator.py, _util.py, and resource_manager.py, using the reported cudaMalloc failure as the first checkpoint. Done means the model hosts successfully under the documented 8-GPU setup without an out-of-memory error.