[Usage]: How to perform Wide Expert Parallelism (Wide EP) inference testing for Qwen3-235B-A22B-FP8 on 8 GPUs using TensorRT LLM?

System Info

How to perform Wide Expert Parallelism (Wide EP) inference testing for Qwen3-235B-A22B-FP8 on 8 GPUs using TensorRT LLM?
Hello everyone, first of all, I would like to express sincere gratitude to the developers of TensorRT LLM—your work has brought incredible value to the community. I have two servers (each equipped with 8 × H20 GPUs). I aim to first conduct Wide EP inference testing for Qwen3-235B-A22B-FP8 and DeepSeek-R1-671B on one server.
Background Information:

root@e86133b171db:/alanscripts# pip show tensorrt_llm
Name: tensorrt_llm
Version: 1.2.0rc4
Summary: TensorRT LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and supports state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs.
Home-page: https://github.com/NVIDIA/TensorRT-LLM
Author: NVIDIA Corporation
Author-email:
License: Apache License 2.0
Location: /usr/local/lib/python3.12/dist-packages
Requires: accelerate, aenum, backoff, blake3, blobfile, build, click, click_option_group, colored, cuda-python, datasets, diffusers, einops, etcd3, evaluate, fastapi, flashinfer-python, h5py, jsonschema, lark, llguidance, matplotlib, meson, mpi4py, mpmath, ninja, numexpr, numpy, nvidia-cuda-nvrtc, nvidia-cutlass-dsl, nvidia-ml-py, nvidia-modelopt, nvidia-nccl-cu13, nvtx, omegaconf, onnx, onnx_graphsurgeon, openai, openai-harmony, opencv-python-headless, optimum, ordered-set, pandas, partial_json_parser, patchelf, peft, pillow, plotly, polygraphy, prometheus_client, prometheus_fastapi_instrumentator, psutil, pulp, pydantic, pydantic-settings, pyzmq, sentencepiece, setuptools, soundfile, starlette, StrEnum, tensorrt, tiktoken, torch, torchvision, transformers, triton, uvicorn, wheel, xgrammar
Required-by:

Inference Service Script (Executed Inside the Container):

torchrun \
  --nproc_per_node 8 \
  /usr/local/bin/trtllm-serve \
  serve \
  /data3/Qwen3-235B-A22B-FP8 \
  --tp_size 2 \
  --pp_size 1 \
  --ep_size 4 \
  --max_seq_len 1024 \
  --max_num_tokens 16384 \
  --port 8000 \
  --max_batch_size 1 \
  --tokenizer /data3/Qwen3-235B-A22B-FP8 \
  --host 0.0.0.0 \
  --trust_remote_code

However, this script has issues. Additionally, I am unsure how to write an inference test script that supports concurrency levels = 10, 20, ..., 100.
If possible, I would greatly appreciate your help and guidance. Thank you!

How would you like to use TensorRT-LLM

I want to run inference of a [specific model](put Hugging Face link here). I don't know how to integrate it with TensorRT-LLM or optimize it for my use case.

Specific questions:

• Model:
• Use case (e.g., chatbot, batch inference, real-time serving):
• Expected throughput/latency requirements:
• Multi-GPU setup needed:

Before submitting a new issue...

• Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.