trtllm-serve - not enough slots available in kaggle notebook

System Info

cuda_architecture sm_75 / x86_64
gpu name 2 x T4
TensorRT-LLM version: 1.0.0rc1

I am setting up a demo on kaggle to show trtllm speculative decoding. I have two GPU, and initiate the model, like,

export LD_LIBRARY_PATH="/kaggle/usr/lib/tensortt_llm_cp311_utility_part1/nvidia/nvjitlink/lib/:$LD_LIBRARY_PATH" && \
trtllm-serve  /kaggle/input/tensorrt-llm-openmath-nemotron-7b-int8-trtllm/blog/OpenMath-Nemotron-7B-int8-trtllm/  \
   --tp_size 2     --kv_cache_free_gpu_memory_fraction 0.92   \
   --max_batch_size 8     --max_num_tokens 1024 

I have two gpus available in the notebook, and get the message, start MpiSession with 2 workers ... but the hosting fails with There are not enough slots available in the system to satisfy the 2 slots.

Full logs of the error here.

I have tried a number of different ways with explicitely using mpirun and the --oversubscribe flags.

I understand trtllm-serve uses and OpenAIServer, this is working with vllm in the same environment - example here.

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

Copy and edit notebook in kaggle, the run the notebook,

Expected behavior

trtllm-serve hosts model

actual behavior

trtllm-serve command errors.

additional notes

n/a