Error in mpi4py when using official sanity check code from TensorRT-LLM v0.19.0rc0 on NVIDIA 5090

• x86_64
• GPU: NVIDIA RTX 5090 32GB
• TensorRT-LLM version: v0.19.0rc0
• CUDA 12.8
• NVIDIA Driver Version 570.124.06

Reproduce progress:
1 .Install tensorrt-llm via the official installation guide with version v0.19.0rc0 since it supports CUDA 12.8 on 5090:

sudo apt-get -y install libopenmpi-dev && pip3 install --upgrade pip setuptools && pip3 install tensorrt_llm==v0.19.0rc0 --extra-index-url https://pypi.nvidia.com

2. execute the following official Sanity check code

from tensorrt_llm import LLM, SamplingParams


def main():

    prompts = [
        "Hello, my name is",
        "The president of the United States is",
        "The capital of France is",
        "The future of AI is",
    ]
    sampling_params = SamplingParams(temperature=0.8, top_p=0.95)

    llm = LLM(model="TinyLlama/TinyLlama-1.1B-Chat-v1.0")

    outputs = llm.generate(prompts, sampling_params)

    # Print the outputs.
    for output in outputs:
        prompt = output.prompt
        generated_text = output.outputs[0].text
        print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")


# The entry point of the program need to be protected for spawning processes.
if __name__ == '__main__':
    main()

Where I got this error:

Traceback (most recent call last):
  File "/home/user/code/sanity_check.py", line 1, in <module>
    from tensorrt_llm import LLM, SamplingParams
  File "/home/user/miniconda3/envs/trt/lib/python3.10/site-packages/tensorrt_llm/__init__.py", line 32, in <module>
    import tensorrt_llm.functional as functional
  File "/home/user/miniconda3/envs/trt/lib/python3.10/site-packages/tensorrt_llm/functional.py", line 28, in <module>
    from . import graph_rewriting as gw
  File "/home/user/miniconda3/envs/trt/lib/python3.10/site-packages/tensorrt_llm/graph_rewriting.py", line 11, in <module>
    from ._utils import trt_gte
  File "/home/user/miniconda3/envs/trt/lib/python3.10/site-packages/tensorrt_llm/_utils.py", line 475, in <module>
    local_comm = mpi_comm().Split_type(split_type=OMPI_COMM_TYPE_HOST)
  File "src/mpi4py/MPI.src/Comm.pyx", line 233, in mpi4py.MPI.Comm.Split_type
mpi4py.MPI.Exception: Invalid argument, error stack:
internal_Comm_split_type(102): MPI_Comm_split_type(MPI_COMM_WORLD, split_type=9, key=0, MPI_INFO_NULL, newcomm=0x79de4219d780) failed
internal_Comm_split_type(74).: Invalid split_type argument (9)

The versions of related package are as follows:

tensorrt                 10.8.0.43
tensorrt_cu12            10.8.0.43
tensorrt_cu12_bindings   10.8.0.43
tensorrt_cu12_libs       10.8.0.43
tensorrt-llm             0.19.0rc0
mpi4py                   4.0.3

Start by reproducing the official TensorRT-LLM v0.19.0rc0 sanity check with the listed CUDA, driver, TensorRT, and mpi4py versions. Inspect tensorrt_llm/_utils.py around line 475, where mpi_comm().Split_type is called, and compare the observed MPI error with the supported mpi4py/Open MPI environment; done would require a confirmed cause and a documented or tested resolution.