TensorRT-LLM - Stub process `multimodal_encoders_0_0` is not healthy
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
TensorRT-LLM - Stub process multimodal_encoders_0_0 is not healthy
Hi all,
I recently try to run Llama 3.2 11B Vision on NVIDIA Triton, I follow this docs step-by-step https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/tensorrtllm_backend/docs/multimodal.html to run the LLM model, but unfortunately when I run the scripts/launch_triton_server.py the model multimodal_encoders is not healthy. Bellow is the details. If you have any suggestion/solution please inform to me.
Thank you,
This is the details:
Env
- Using docker image :
nvcr.io/nvidia/tritonserver:25.03-trtllm-python-py3 - 1GPU Nvidia A100 with 40 Gb GPU Memory
- PIP List :
Package Version
---------------------- -----------------------------
accelerate 1.6.0
aenum 3.1.15
aiohappyeyeballs 2.6.1
aiohttp 3.11.16
aiosignal 1.3.2
annotated-types 0.7.0
anyio 4.9.0
attrs 25.3.0
blinker 1.7.0
build 1.2.2.post1
certifi 2025.1.31
charset-normalizer 3.4.1
click 8.1.8
click-option-group 0.5.7
cloudpickle 3.1.1
colored 2.3.0
cryptography 41.0.7
cuda-bindings 12.8.0
cuda-python 12.8.0
cupy-cuda12x 13.4.1
datasets 3.5.0
dbus-python 1.3.2
diffusers 0.32.2
dill 0.3.8
distlib 0.3.9
distro 1.9.0
einops 0.8.1
evaluate 0.4.3
fastapi 0.115.6
fastrlock 0.8.3
filelock 3.18.0
flashinfer-python 0.2.4
frozenlist 1.5.0
fsspec 2024.12.0
grpcio 1.71.0
grpcio-tools 1.64.0
h11 0.14.0
h5py 3.12.1
httpcore 1.0.7
httplib2 0.20.4
httpx 0.27.2
huggingface-hub 0.30.1
idna 3.10
importlib_metadata 8.6.1
iniconfig 2.1.0
Jinja2 3.1.6
jiter 0.9.0
lark 1.2.2
launchpadlib 1.11.0
lazr.restfulclient 0.14.6
lazr.uri 1.0.6
markdown-it-py 3.0.0
MarkupSafe 3.0.2
mdurl 0.1.2
mpi4py 3.1.5
mpmath 1.3.0
multidict 6.3.1
multiprocess 0.70.16
networkx 3.4.2
ninja 1.11.1.4
numpy 1.26.4
nvidia-cuda-nvrtc-cu12 12.8.93
nvidia-ml-py 12.570.86
nvidia-modelopt 0.25.0
nvidia-modelopt-core 0.25.0
nvidia-nccl-cu12 2.26.2
oauthlib 3.2.2
onnx 1.16.1
onnx-graphsurgeon 0.5.7
openai 1.60.0
optimum 1.24.0
ordered-set 4.1.0
packaging 23.2
pandas 2.2.3
pillow 10.3.0
pip 24.0
platformdirs 4.3.7
pluggy 1.5.0
polygraphy 0.49.9
propcache 0.3.1
protobuf 5.29.4
psutil 7.0.0
PuLP 3.1.1
pyarrow 19.0.1
pydantic 2.11.1
pydantic_core 2.33.0
Pygments 2.19.1
PyGObject 3.48.2
PyJWT 2.7.0
pynvml 12.0.0
pyparsing 3.1.1
pyproject_hooks 1.2.0
pytest 8.3.5
python-apt 2.7.7+ubuntu4
python-dateutil 2.9.0.post0
pytorch-triton 3.2.0+gitb2684bf3b.nvinternal
pytz 2025.2
PyYAML 6.0.2
pyzmq 26.3.0
regex 2024.11.6
requests 2.32.3
rich 14.0.0
safetensors 0.5.3
scipy 1.15.2
sentencepiece 0.2.0
setuptools 68.1.2
six 1.17.0
sniffio 1.3.1
starlette 0.41.3
StrEnum 0.4.15
sympy 1.13.1
tensorrt 10.9.0.34
tensorrt_llm 0.18.0
tokenizers 0.21.1
torch 2.7.0a0+7c8ec84dab.nv25.3
torchprofile 0.0.4
torchvision 0.22.0a0
tqdm 4.67.1
transformers 4.47.1
tritonfrontend 2.56.0
tritonserver 0.0.0
typing_extensions 4.13.0
typing-inspection 0.4.0
tzdata 2025.2
urllib3 2.3.0
uvicorn 0.34.0
virtualenv 20.30.0
wadllib 1.3.6
wheel 0.42.0
xxhash 3.5.0
yarl 1.18.3
zipp 3.21.0
- Log From Build TensorRT-LLM engines
convert_checkpoint.py
root@llm-experiment-instance:/opt/tritonserver/model_repository/tensorrtllm_backend# python tensorrt_llm/examples/mllama/convert_checkpoint.py \
--model_dir ${HF_MODEL_PATH} \
--output_dir ${UNIFIED_CKPT_PATH} \
--dtype bfloat16
2025-04-17 06:56:37,484 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT-LLM version: 0.18.0
0.18.0
318it [00:00, 1089.85it/s]
Total time of reading and converting 0.5500814914703369 s
Total time of saving checkpoint 114.87693428993225 s
Total time of converting checkpoints: 00:01:56
trtllm-build
root@llm-experiment-instance:/opt/tritonserver/model_repository/tensorrtllm_backend# trtllm-build \
--checkpoint_dir ${UNIFIED_CKPT_PATH} \
--output_dir ${ENGINE_PATH} \
--gemm_plugin auto \
--max_batch_size 8 \
--max_seq_len 2048 \
--max_num_tokens 4096 \
--max_encoder_input_len 6404
2025-04-17 07:07:55,838 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT-LLM version: 0.18.0
[04/17/2025-07:07:55] [TRT-LLM] [I] Set bert_attention_plugin to auto.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set gpt_attention_plugin to auto.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set gemm_plugin to auto.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set gemm_swiglu_plugin to None.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set fp8_rowwise_gemm_plugin to None.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set nccl_plugin to auto.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set lora_plugin to None.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set moe_plugin to auto.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set mamba_conv1d_plugin to auto.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set low_latency_gemm_plugin to None.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set low_latency_gemm_swiglu_plugin to None.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set context_fmha to True.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set bert_context_fmha_fp32_acc to False.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set remove_input_padding to True.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set reduce_fusion to False.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set user_buffer to False.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set tokens_per_block to 64.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set use_paged_context_fmha to False.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set use_fp8_context_fmha to False.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set multiple_profiles to False.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set paged_state to True.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set streamingllm to False.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set use_fused_mlp to True.
[04/17/2025-07:07:55] [TRT-LLM] [I] Set pp_reduce_scatter to False.
[04/17/2025-07:07:55] [TRT-LLM] [W] Implicitly setting MLLaMAConfig.skip_cross_attn_blocks = True
[04/17/2025-07:07:56] [TRT-LLM] [W] Provided but not required tensors: {'embed_positions_for_gpt_attention', 'rotary_inv_freq', 'embed_positions'}
[04/17/2025-07:07:56] [TRT-LLM] [I] Set dtype to bfloat16.
[04/17/2025-07:07:56] [TRT-LLM] [I] Set paged_kv_cache to True.
[04/17/2025-07:07:56] [TRT-LLM] [W] Overriding paged_state to False
[04/17/2025-07:07:56] [TRT-LLM] [I] Set paged_state to False.
[04/17/2025-07:07:56] [TRT-LLM] [W] remove_input_padding is enabled, while opt_num_tokens is not set, setting to max_batch_size*max_beam_width.
[04/17/2025-07:07:56] [TRT-LLM] [W] padding removal and fMHA are both enabled, max_input_len is not required and will be ignored
[04/17/2025-07:08:00] [TRT] [I] [MemUsageChange] Init CUDA: CPU +17, GPU +0, now: CPU 9177, GPU 426 (MiB)
[04/17/2025-07:08:04] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +2486, GPU +382, now: CPU 11864, GPU 808 (MiB)
[04/17/2025-07:08:04] [TRT-LLM] [I] Set nccl_plugin to None.
[04/17/2025-07:08:05] [TRT-LLM] [I] Total time of constructing network from module object 9.355340957641602 seconds
[04/17/2025-07:08:05] [TRT-LLM] [I] Total optimization profiles added: 1
[04/17/2025-07:08:05] [TRT-LLM] [I] Total time to initialize the weights in network Unnamed Network 0: 00:00:00
[04/17/2025-07:08:05] [TRT-LLM] [I] Build TensorRT engine Unnamed Network 0
[04/17/2025-07:08:05] [TRT] [W] Unused Input: host_kv_cache_block_offsets
[04/17/2025-07:08:05] [TRT] [W] Unused Input: cross_kv_cache_gen
[04/17/2025-07:08:05] [TRT] [W] [RemoveDeadLayers] Input Tensor host_kv_cache_block_offsets is unused or used only at compile-time, but is not being removed.
[04/17/2025-07:08:05] [TRT] [W] [RemoveDeadLayers] Input Tensor cross_kv_cache_gen is unused or used only at compile-time, but is not being removed.
[04/17/2025-07:08:05] [TRT] [I] Global timing cache in use. Profiling results in this builder pass will be stored.
[04/17/2025-07:08:05] [TRT] [I] Compiler backend is used during engine build.
[04/17/2025-07:08:16] [TRT] [E] Error Code: 9: Skipping tactic 0x00000000000003ea due to exception Unsupported data type Bool.
[04/17/2025-07:08:16] [TRT] [I] [GraphReduction] The approximate region cut reduction algorithm is called.
[04/17/2025-07:08:16] [TRT] [I] Detected 27 inputs and 1 output network tensors.
[04/17/2025-07:08:30] [TRT] [I] Total Host Persistent Memory: 131264 bytes
[04/17/2025-07:08:30] [TRT] [I] Total Device Persistent Memory: 0 bytes
[04/17/2025-07:08:30] [TRT] [I] Max Scratch Memory: 209846272 bytes
[04/17/2025-07:08:30] [TRT] [I] [BlockAssignment] Started assigning block shifts. This will take 684 steps to complete.
[04/17/2025-07:08:30] [TRT] [I] [BlockAssignment] Algorithm ShiftNTopDown took 164.485ms to assign 31 blocks to 684 nodes requiring 1302522880 bytes.
[04/17/2025-07:08:30] [TRT] [I] Total Activation Memory: 1302520832 bytes
[04/17/2025-07:08:31] [TRT] [I] Total Weights Memory: 19751711360 bytes
[04/17/2025-07:08:31] [TRT] [I] Compiler backend is used during engine execution.
[04/17/2025-07:08:31] [TRT] [I] Engine generation completed in 25.3221 seconds.
[04/17/2025-07:08:31] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 0 MiB, GPU 18837 MiB
[04/17/2025-07:08:43] [TRT-LLM] [I] Total time of building Unnamed Network 0: 00:00:37
[04/17/2025-07:08:43] [TRT] [I] Serialized 27 bytes of code generator cache.
[04/17/2025-07:08:43] [TRT] [I] Serialized 472500 bytes of compilation cache.
[04/17/2025-07:08:43] [TRT] [I] Serialized 13 timing cache entries
[04/17/2025-07:08:43] [TRT-LLM] [I] Timing cache serialized to model.cache
[04/17/2025-07:08:43] [TRT-LLM] [I] Build phase peak memory: 50856.69 MB, children: 16.81 MB
[04/17/2025-07:08:43] [TRT-LLM] [I] Serializing engine to tmp/trt_engines/Llama-3.2-11B-Vision/fp16/1-gpu/rank0.engine...
[04/17/2025-07:09:37] [TRT-LLM] [I] Engine serialized. Total time: 00:00:53
[04/17/2025-07:09:37] [TRT-LLM] [I] Total time of building all engines: 00:01:42
build_visual_engine (I use build_visual_engine rather than build_multimodal_engine, because from GitHub repo TensorRT-LLM there is only build_visual_engine.py)
root@llm-experiment-instance:/opt/tritonserver/model_repository/tensorrtllm_backend# python tensorrt_llm/examples/multimodal/build_visual_engine.py --model_path ${HF_MODEL_PATH} --m
odel_type mllama --output_dir ${MULTIMODAL_ENGINE_PATH} --max_batch_size 8 #max_batch_size * max_num_images_per_request
2025-04-17 07:13:53,033 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
[TensorRT-LLM] TensorRT-LLM version: 0.18.0
The model weights are not tied. Please use the `tie_weights` method before using the `infer_auto_device` function.
Loading checkpoint shards: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:15<00:00, 3.04s/it]
[04/17/2025-07:14:10] [TRT] [I] Exporting onnx to tmp/trt_engines/Llama-3.2-11B-Vision/multimodal_encoder/visual_encoder/onnx/model.onnx
/usr/local/lib/python3.12/dist-packages/transformers/models/mllama/modeling_mllama.py:1564: TracerWarning: Converting a tensor to a Python boolean might cause the trace to be incorrect. We can't record the data flow of Python values, so this value will be treated as a constant in the future. This means that the trace might not generalize to other inputs! slice_index = -num_padding_patches if num_padding_patches > 0 else None
In-place op on output of tensor.shape. See https://pytorch.org/docs/main/onnx.html#avoid-inplace-operations-when-using-tensor-shape-in-tracing-mode
[04/17/2025-07:14:25] [TRT] [I] Building TRT engine to tmp/trt_engines/Llama-3.2-11B-Vision/multimodal_encoder/visual_encoder.engine
[04/17/2025-07:14:25] [TRT] [I] [MemUsageChange] Init CUDA: CPU -2, GPU +0, now: CPU 876, GPU 27738 (MiB)
[04/17/2025-07:14:37] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +2487, GPU +380, now: CPU 3519, GPU 28118 (MiB)
[04/17/2025-07:14:38] [TRT] [W] ModelImporter.cpp:459: Make sure input aspect_ratio_ids has Int64 binding.
[04/17/2025-07:14:38] [TRT] [W] ModelImporter.cpp:459: Make sure input aspect_ratio_mask has Int64 binding.
[04/17/2025-07:14:38] [TRT] [I] Succeeded parsing tmp/trt_engines/Llama-3.2-11B-Vision/multimodal_encoder/visual_encoder/onnx/model.onnx
[04/17/2025-07:14:39] [TRT] [I] Global timing cache in use. Profiling results in this builder pass will be stored.
[04/17/2025-07:14:39] [TRT] [I] Compiler backend is used during engine build.
[04/17/2025-07:15:34] [TRT] [I] Detected 3 inputs and 1 output network tensors.
[04/17/2025-07:15:36] [TRT] [I] Total Host Persistent Memory: 5856 bytes
[04/17/2025-07:15:36] [TRT] [I] Total Device Persistent Memory: 0 bytes
[04/17/2025-07:15:36] [TRT] [I] Max Scratch Memory: 1894646784 bytes
[04/17/2025-07:15:36] [TRT] [I] [BlockAssignment] Started assigning block shifts. This will take 6 steps to complete.
[04/17/2025-07:15:36] [TRT] [I] [BlockAssignment] Algorithm ShiftNTopDown took 0.022312ms to assign 2 blocks to 6 nodes requiring 1997407232 bytes.
[04/17/2025-07:15:36] [TRT] [I] Total Activation Memory: 1997407232 bytes
[04/17/2025-07:15:36] [TRT] [I] Total Weights Memory: 1738012930 bytes
[04/17/2025-07:15:36] [TRT] [I] Compiler backend is used during engine execution.
[04/17/2025-07:15:36] [TRT] [I] Engine generation completed in 57.6624 seconds.
[04/17/2025-07:15:36] [TRT] [I] [MemUsageStats] Peak memory usage of TRT CPU/GPU memory allocators: CPU 3 MiB, GPU 3797 MiB
[04/17/2025-07:15:37] [TRT] [I] Succeeded building tmp/trt_engines/Llama-3.2-11B-Vision/multimodal_encoder/visual_encoder.engine in 58 s
scripts/launch_triton_server.pyLog
root@llm-experiment-instance:/opt/tritonserver/model_repository/tensorrtllm_backend# python3 scripts/launch_triton_server.py --world_size 1 --model_repo=multimodal_ifb/ --tensorrt_llm_model_name tensorrt_llm,multimodal_encoders --multimodal_gpu0_cuda_mem_pool_bytes 300000000
root@llm-experiment-instance:/opt/tritonserver/model_repository/tensorrtllm_backend# I0417 07:24:38.886677 1939 pinned_memory_manager.cc:277] "Pinned memory pool is created at '0x7faf74000000' with size 268435456"
I0417 07:24:38.889300 1939 cuda_memory_manager.cc:107] "CUDA memory pool is created on device 0 with size 300000000"
I0417 07:24:38.898001 1939 model_lifecycle.cc:473] "loading: multimodal_encoders:1"
I0417 07:24:38.898063 1939 model_lifecycle.cc:473] "loading: postprocessing:1"
I0417 07:24:38.898112 1939 model_lifecycle.cc:473] "loading: preprocessing:1"
I0417 07:24:38.898214 1939 model_lifecycle.cc:473] "loading: tensorrt_llm:1"
I0417 07:24:38.898311 1939 model_lifecycle.cc:473] "loading: tensorrt_llm_bls:1"
I0417 07:24:39.074887 1939 python_be.cc:2249] "TRITONBACKEND_ModelInstanceInitialize: multimodal_encoders_0_0 (CPU device 0)"
I0417 07:24:39.074893 1939 python_be.cc:2249] "TRITONBACKEND_ModelInstanceInitialize: postprocessing_0_0 (CPU device 0)"
I0417 07:24:39.074941 1939 python_be.cc:2249] "TRITONBACKEND_ModelInstanceInitialize: preprocessing_0_0 (CPU device 0)"
I0417 07:24:39.267847 1939 libtensorrtllm.cc:55] "TRITONBACKEND_Initialize: tensorrtllm"
I0417 07:24:39.267910 1939 libtensorrtllm.cc:62] "Triton TRITONBACKEND API version: 1.19"
I0417 07:24:39.267918 1939 libtensorrtllm.cc:66] "'tensorrtllm' TRITONBACKEND API version: 1.19"
I0417 07:24:39.267926 1939 libtensorrtllm.cc:86] "backend configuration:\n{\"cmdline\":{\"auto-complete-config\":\"false\",\"backend-directory\":\"/opt/tritonserver/backends\",\"min-compute-capability\":\"6.000000\",\"default-max-batch-size\":\"4\"}}"
[TensorRT-LLM][WARNING] gpu_device_ids is not specified, will be automatically set
I0417 07:24:39.302509 1939 libtensorrtllm.cc:114] "TRITONBACKEND_ModelInitialize: tensorrt_llm (version 1)"
[TensorRT-LLM][WARNING] participant_ids is not specified, will be automatically set
[TensorRT-LLM][WARNING] iter_stats_max_iterations is not specified, will use default value of 1000
[TensorRT-LLM][WARNING] request_stats_max_iterations is not specified, will use default value of 0
[TensorRT-LLM][WARNING] normalize_log_probs is not specified, will be set to true
[TensorRT-LLM][WARNING] max_tokens_in_paged_kv_cache is not specified, will use default value
[TensorRT-LLM][WARNING] kv_cache_free_gpu_mem_fraction is not specified, will use default value of 0.9 or max_tokens_in_paged_kv_cache
[TensorRT-LLM][WARNING] kv_cache_host_memory_bytes not set, defaulting to 0
[TensorRT-LLM][WARNING] kv_cache_onboard_blocks not set, defaulting to true
[TensorRT-LLM][WARNING] max_attention_window_size is not specified, will use default value (i.e. max_sequence_length)
[TensorRT-LLM][WARNING] sink_token_length is not specified, will use default value
[TensorRT-LLM][WARNING] batch_scheduler_policy parameter was not found or is invalid (must be max_utilization or guaranteed_no_evict)
[TensorRT-LLM][WARNING] lora_cache_max_adapter_size not set, defaulting to 64
[TensorRT-LLM][WARNING] lora_cache_optimal_adapter_size not set, defaulting to 8
[TensorRT-LLM][WARNING] lora_cache_gpu_memory_fraction not set, defaulting to 0.05
[TensorRT-LLM][WARNING] lora_cache_host_memory_bytes not set, defaulting to 1GB
[TensorRT-LLM][WARNING] multi_block_mode is not specified, will be set to true
[TensorRT-LLM][WARNING] enable_context_fmha_fp32_acc is not specified, will be set to false
[TensorRT-LLM][WARNING] cuda_graph_mode is not specified, will be set to false
[TensorRT-LLM][WARNING] cuda_graph_cache_size is not specified, will be set to 0
[TensorRT-LLM][INFO] speculative_decoding_fast_logits is not specified, will be set to false
[TensorRT-LLM][WARNING] decoding_mode parameter is invalid or not specified(must be one of the {top_k, top_p, top_k_top_p, beam_search, medusa, redrafter, lookahead, eagle}).Using default: top_k_top_p if max_beam_width == 1, beam_search otherwise
[TensorRT-LLM][WARNING] gpu_weights_percent parameter is not specified, will use default value of 1.0
[TensorRT-LLM][INFO] recv_poll_period_ms is not set, will use busy loop
[TensorRT-LLM][WARNING] encoder_model_path is not specified, will be left empty
[TensorRT-LLM][INFO] Engine version 0.18.0 found in the config file, assuming engine(s) built by new builder API.
[TensorRT-LLM][INFO] Setting encoder max input length and hidden size for accepting visual features.
[TensorRT-LLM][INFO] Initializing MPI with thread mode 3
[TensorRT-LLM][INFO] Initialized MPI
[TensorRT-LLM][INFO] Refreshed the MPI local session
[TensorRT-LLM][INFO] MPI size: 1, MPI local size: 1, rank: 0
[TensorRT-LLM][INFO] Rank 0 is using GPU 0
[TensorRT-LLM][INFO] TRTGptModel maxNumSequences: 8
[TensorRT-LLM][INFO] TRTGptModel maxBatchSize: 8
[TensorRT-LLM][INFO] TRTGptModel maxBeamWidth: 1
[TensorRT-LLM][INFO] TRTGptModel maxSequenceLen: 2048
[TensorRT-LLM][INFO] TRTGptModel maxDraftLen: 0
[TensorRT-LLM][INFO] TRTGptModel mMaxAttentionWindowSize: (2048) * 40
[TensorRT-LLM][INFO] TRTGptModel enableTrtOverlap: 0
[TensorRT-LLM][INFO] TRTGptModel normalizeLogProbs: 1
[TensorRT-LLM][INFO] TRTGptModel maxNumTokens: 4096
[TensorRT-LLM][INFO] TRTGptModel maxInputLen: 2047 = min(maxSequenceLen - 1, maxNumTokens) since context FMHA and usePackedInput are enabled
[TensorRT-LLM][INFO] TRTGptModel If model type is encoder, maxInputLen would be reset in trtEncoderModel to maxInputLen: min(maxSequenceLen, maxNumTokens).
[TensorRT-LLM][INFO] Capacity Scheduler Policy: GUARANTEED_NO_EVICT
[TensorRT-LLM][INFO] Context Chunking Scheduler Policy: None
[TensorRT-LLM][WARNING] Don't setup 'skip_special_tokens' correctly (set value is ${skip_special_tokens}). Set it as True by default.
[TensorRT-LLM][WARNING] Don't setup 'add_special_tokens' correctly (set value is ${add_special_tokens}). Set it as True by default.
I0417 07:24:43.912028 1939 model_lifecycle.cc:849] "successfully loaded 'postprocessing'"
I0417 07:24:43.918493 1939 python_be.cc:2249] "TRITONBACKEND_ModelInstanceInitialize: tensorrt_llm_bls_0_0 (CPU device 0)"
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):
getting local rank failed
--> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like orte_init failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during orte_init; some of which are due to configuration or
environment problems. This failure appears to be an internal failure;
here's some additional information (which may only be relevant to an
Open MPI developer):
orte_ess_init failed
--> Returned value No permission (-17) instead of ORTE_SUCCESS
--------------------------------------------------------------------------
--------------------------------------------------------------------------
It looks like MPI_INIT failed for some reason; your parallel process is
likely to abort. There are many reasons that a parallel process can
fail during MPI_INIT; some of which are due to configuration or environment
problems. This failure appears to be an internal failure; here's some
additional information (which may only be relevant to an Open MPI
developer):
ompi_mpi_init: ompi_rte_init failed
--> Returned "No permission" (-17) instead of "Success" (0)
--------------------------------------------------------------------------
*** An error occurred in MPI_Init_thread
*** on a NULL communicator
*** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
*** and potentially your MPI job)
[llm-experiment-instance:01961] Local abort before MPI_INIT completed completed successfully, but am not able to aggregate error messages, and not able to guarantee that all other processes were killed!
I0417 07:24:45.775356 1939 model_lifecycle.cc:849] "successfully loaded 'preprocessing'"
I0417 07:24:46.166780 1939 model_lifecycle.cc:849] "successfully loaded 'tensorrt_llm_bls'"
E0417 07:24:47.139419 1939 backend_model.cc:692] "ERROR: Failed to create instance: Stub process 'multimodal_encoders_0_0' is not healthy."
E0417 07:24:47.139566 1939 model_lifecycle.cc:654] "failed to load 'multimodal_encoders' version 1: Internal: Stub process 'multimodal_encoders_0_0' is not healthy."
I0417 07:24:47.139589 1939 model_lifecycle.cc:789] "failed to load 'multimodal_encoders'"
[TensorRT-LLM][INFO] Loaded engine size: 18848 MiB
[TensorRT-LLM][INFO] Inspecting the engine to identify potential runtime issues...
[TensorRT-LLM][INFO] The profiling verbosity of the engine does not allow this analysis to proceed. Re-build the engine with 'detailed' profiling verbosity to get more diagnostics.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 1242.18 MiB for execution context memory.
[TensorRT-LLM][INFO] [MS] Running engine with multi stream info
[TensorRT-LLM][INFO] [MS] Number of aux streams is 1
[TensorRT-LLM][INFO] [MS] Number of total worker streams is 2
[TensorRT-LLM][INFO] [MS] The main stream provided by execute/enqueue calls is the first worker stream
[TensorRT-LLM][INFO] [MemUsageChange] TensorRT-managed allocation in IExecutionContext creation: CPU +0, GPU +0, now: CPU 0, GPU 18836 (MiB)
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 404.59 MB GPU memory for runtime buffers.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 16.41 MB GPU memory for decoder.
[TensorRT-LLM][INFO] Memory usage when calculating max tokens in paged kv cache: total: 39.38 GiB, available: 18.55 GiB
[TensorRT-LLM][INFO] Number of blocks in KV cache primary pool: 2137
[TensorRT-LLM][INFO] Number of blocks in KV cache secondary pool: 0, onboard blocks to primary memory before reuse: true
[TensorRT-LLM][INFO] Number of tokens per block: 64.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 8.34 GiB for max tokens in paged KV cache (68352).
[TensorRT-LLM][INFO] Number of tokens per block: 64.
[TensorRT-LLM][INFO] [MemUsageChange] Allocated 8.35 GiB for max tokens in paged KV cache (273536).
[TensorRT-LLM][INFO] This is an Encoder-Decoder model, set 0.5 cross KV cache fraction based on the config.
[TensorRT-LLM][INFO] Number of blocks in self KV cache primary pool: 1068, in cross KV cache primary pool: 4274
[TensorRT-LLM][INFO] Number of blocks in self KV cache secondary pool: 0, in cross KV cache secondary pool: 0
[TensorRT-LLM][WARNING] exclude_input_in_output is not specified, will be set to false
[TensorRT-LLM][WARNING] cancellation_check_period_ms is not specified, will be set to 100 (ms)
[TensorRT-LLM][WARNING] stats_check_period_ms is not specified, will be set to 100 (ms)
I0417 07:24:52.611352 1939 libtensorrtllm.cc:184] "TRITONBACKEND_ModelInstanceInitialize: tensorrt_llm_0_0"
I0417 07:24:52.612245 1939 model_lifecycle.cc:849] "successfully loaded 'tensorrt_llm'"
E0417 07:24:52.612340 1939 model_repository_manager.cc:703] "Invalid argument: ensemble 'ensemble' depends on 'multimodal_encoders' which has no loaded version. Model 'multimodal_encoders' loading failed with error: version 1 is at UNAVAILABLE state: Internal: Stub process 'multimodal_encoders_0_0' is not healthy.;"
I0417 07:24:52.612445 1939 server.cc:604]
+------------------+------+
| Repository Agent | Path |
+------------------+------+
+------------------+------+
I0417 07:24:52.612483 1939 server.cc:631]
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| Backend | Path | Config |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
| python | /opt/tritonserver/backends/python/libtriton_python.so | {"cmdline":{"auto-complete-config":"false","backend-directory":"/opt/tritonserver/backends","min- |
| | | compute-capability":"6.000000","shm-region-prefix-name":"prefix0_","default-max-batch-size":"4"}} |
| tensorrtllm | /opt/tritonserver/backends/tensorrtllm/libtriton_tensorrtllm.so | {"cmdline":{"auto-complete-config":"false","backend-directory":"/opt/tritonserver/backends","min- |
| | | compute-capability":"6.000000","default-max-batch-size":"4"}} |
+-------------+-----------------------------------------------------------------+---------------------------------------------------------------------------------------------------+
I0417 07:24:52.612575 1939 server.cc:674]
+---------------------+---------+-------------------------------------------------------------------------------+
| Model | Version | Status |
+---------------------+---------+-------------------------------------------------------------------------------+
| multimodal_encoders | 1 | UNAVAILABLE: Internal: Stub process 'multimodal_encoders_0_0' is not healthy. |
| postprocessing | 1 | READY |
| preprocessing | 1 | READY |
| tensorrt_llm | 1 | READY |
| tensorrt_llm_bls | 1 | READY |
+---------------------+---------+-------------------------------------------------------------------------------+
I0417 07:24:52.660320 1939 metrics.cc:890] "Collecting metrics for GPU 0: NVIDIA A100-SXM4-40GB"
I0417 07:24:52.666527 1939 metrics.cc:783] "Collecting CPU metrics"
I0417 07:24:52.666712 1939 tritonserver.cc:2598]
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| Option | Value |
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
| server_id | triton |
| server_version | 2.56.0 |
| server_extensions | classification sequence model_repository model_repository(unload_dependents) schedule_policy model_configuration system_shared_memory cuda_sha |
| | red_memory binary_tensor_data parameters statistics trace logging |
| model_repository_path[0] | multimodal_ifb/ |
| model_control_mode | MODE_NONE |
| strict_model_config | 1 |
| model_config_name | |
| rate_limit | OFF |
| pinned_memory_pool_byte_size | 268435456 |
| cuda_memory_pool_byte_size{0} | 300000000 |
| min_supported_compute_capability | 6.0 |
| strict_readiness | 1 |
| exit_timeout | 30 |
| cache_enabled | 0 |
+----------------------------------+------------------------------------------------------------------------------------------------------------------------------------------------+
I0417 07:24:52.666785 1939 server.cc:305] "Waiting for in-flight requests to complete."
I0417 07:24:52.666802 1939 server.cc:321] "Timeout 30: Found 0 model versions that have in-flight inferences"
I0417 07:24:52.667824 1939 server.cc:336] "All models are stopped, unloading models"
I0417 07:24:52.667861 1939 server.cc:345] "Timeout 30: Found 4 live models and 0 in-flight non-inference requests"
[TensorRT-LLM][INFO] Refreshed the MPI local session
I0417 07:24:52.920586 1939 model_lifecycle.cc:636] "successfully unloaded 'tensorrt_llm' version 1"
I0417 07:24:53.668002 1939 server.cc:345] "Timeout 29: Found 3 live models and 0 in-flight non-inference requests"
Cleaning up...
Cleaning up...
Cleaning up...
I0417 07:24:54.300562 1939 model_lifecycle.cc:636] "successfully unloaded 'tensorrt_llm_bls' version 1"
I0417 07:24:54.651181 1939 model_lifecycle.cc:636] "successfully unloaded 'postprocessing' version 1"
I0417 07:24:54.668172 1939 server.cc:345] "Timeout 28: Found 1 live models and 0 in-flight non-inference requests"
I0417 07:24:54.725234 1939 model_lifecycle.cc:636] "successfully unloaded 'preprocessing' version 1"
I0417 07:24:55.668330 1939 server.cc:345] "Timeout 27: Found 0 live models and 0 in-flight non-inference requests"
error: creating server: Internal - failed to load all models
- Folder structure
root@llm-experiment-instance:/opt/tritonserver/model_repository/tensorrtllm_backend/multimodal_ifb# tree -L 3
.
|-- ensemble
| |-- 1
| `-- config.pbtxt
|-- multimodal_encoders
| |-- 1
| | |-- __pycache__
| | |-- model.py
| | `-- multimodal_utils.py
| `-- config.pbtxt
|-- postprocessing
| |-- 1
| | |-- __pycache__
| | `-- model.py
| `-- config.pbtxt
|-- preprocessing
| |-- 1
| | |-- __pycache__
| | `-- model.py
| `-- config.pbtxt
|-- tensorrt_llm
| |-- 1
| | `-- model.py
| `-- config.pbtxt
`-- tensorrt_llm_bls
|-- 1
| |-- __pycache__
| |-- lib
| `-- model.py
`-- config.pbtxt
18 directories, 12 files
root@llm-experiment-instance:/opt/tritonserver/model_repository/tensorrtllm_backend# tree -L 2
.
|-- LICENSE
|-- README.md
|-- all_models
| |-- disaggregated_serving
| |-- gpt
| |-- inflight_batcher_llm
| |-- multimodal
| |-- tests
| `-- whisper
|-- build.sh
|-- ci
| |-- L0_backend_trtllm
| `-- README.md
|-- dockerfile
| |-- Dockerfile.triton.trt_llm_backend
| `-- Dockerfile.trt_llm_backend
|-- docs
| |-- baichuan.md
| |-- build.md
| |-- encoder_decoder.md
| |-- gemma.md
| |-- guided_decoding.md
| |-- llama.md
| |-- llama_multi_instance.md
| |-- lora.md
| |-- model_config.md
| |-- multimodal.md
| `-- whisper.md
|-- images
| |-- leader-mode.png
| `-- orchestrator-mode.png
|-- inflight_batcher_llm
| |-- CMakeLists.txt
| |-- client
| |-- cmake
| |-- scripts
| `-- src
|-- model.cache
|-- multimodal_ifb
| |-- ensemble
| |-- multimodal_encoders
| |-- postprocessing
| |-- preprocessing
| |-- tensorrt_llm
| `-- tensorrt_llm_bls
|-- requirements.txt
|-- scripts
| `-- launch_triton_server.py
|-- tensorrt_llm
| |-- 3rdparty
| |-- CODE_OF_CONDUCT.md
| |-- CODING_GUIDELINES.md
| |-- CONTRIBUTING.md
| |-- LICENSE
| |-- README.md
| |-- benchmarks
| |-- cpp
| |-- docker
| |-- docs
| |-- examples
| |-- jenkins
| |-- pyproject.toml
| |-- requirements-dev.txt
| |-- requirements.txt
| |-- scripts
| |-- setup.py
| |-- tensorrt_llm
| `-- tests
|-- tmp
| |-- hf_models
| |-- trt_engines
| `-- trt_models
`-- tools
|-- __init__.py
|-- fill_template.py
|-- gpt
|-- inflight_batcher_llm
|-- multimodal
|-- tests
|-- utils
|-- utils.sh
|-- version.txt
`-- whisper
48 directories, 36 files
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the multimodal deployment documentation and the commands in the issue, especially scripts/launch_triton_server.py and tensorrt_llm/examples/multimodal/build_visual_engine.py. Reproduce the unhealthy multimodal_encoders process in the stated container and inspect its server logs alongside the engine-build output. Done means the multimodal_encoders stub reports healthy and the documented Llama 3.2 Vision launch completes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python, pytorch
- Domain
- ai, backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100