[Bug]: Bus Error and Crash When Processing Image Requests in trtllm-serve
Open
@yechank-nvidia is already working on this.
Since Aug 29, 2025.
bug
Inference runtime
LLM API
Multimodal
OpenAI API
- Dominant language
- Python
- Stars
- 14.7k
- Forks
- 2.8k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 489
Description
System Info
TensorRT-LLM version: 1.1.0rc1
GPU: A800-SXM4-40GB
Model: Qwen2.5-VL-7B-Instruct
Who can help?
No response
Information
- The official example scripts
- My own modified scripts
Tasks
- An officially supported task in the
examplesfolder (such as GLUE/SQuAD, ...) - My own task or dataset (give details below)
Reproduction
When starting the trtllm-serve service with a Qwen2.5-VL-7B-Instruct model and sending a curl request containing an image URL, the service crashes with a bus error (signal 7) and nonexistent physical address error. This issue occurs in TensorRT-LLM v1.1.0rc1 but not in v1.0.0rc6.
Steps to Reproduce:
1、Start the trtllm-serve service with the following command:
trtllm-serve /data1/llm-models/qwen/Qwen2___5-VL-7B-Instruct \
--extra_llm_api_options ./extra-llm-api-config.yml \
--max_batch_size 10 \
--max_num_tokens 40960 \
--max_seq_len 8192 \
--trust_remote_code
2、Send a curl request to the service:
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "/data1/llm-models/qwen/Qwen2___5-VL-7B-Instruct",
"messages": [
{
"role": "user",
"content": [
{"type": "text", "text": "Tell me what you see in the imags."},
{"type": "image_url", "image_url": {"url": "/data1/yating/qwen2.5_vl_test/demo.jpeg"}}
]
}
],
"max_tokens": 100
}'
3、The service crashes with a bus error and produces a core dump.
Expected behavior
The service should process the image request successfully and return a valid response without crashing.
actual behavior
The service crashes with the following error output:
/usr/local/lib/python3.12/dist-packages/torch/multiprocessing/reductions.py:598: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
if storage.is_cuda:
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
- Avoid using `tokenizers` before the fork if possible
- Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
[e41b8682b5c3:728 :0:1241] Caught signal 7 (Bus error: nonexistent physical address)
[e41b8682b5c3:728 :1:1251] Caught signal 7 (Bus error: nonexistent physical address)
[e41b8682b5c3:728 :2:1219] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723371] [e41b8682b5c3:728 :3] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :3:1235] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723372] [e41b8682b5c3:728 :4] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :4:1261] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723432] [e41b8682b5c3:728 :4] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723372] [e41b8682b5c3:728 :5] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :5:1229] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723372] [e41b8682b5c3:728 :6] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :6:1216] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723487] [e41b8682b5c3:728 :6] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723382] [e41b8682b5c3:728 :7] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 4 was not set in ucs
[e41b8682b5c3:728 :7:1209] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723518] [e41b8682b5c3:728 :7] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723518] [e41b8682b5c3:728 :10] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :10:1253] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723391] [e41b8682b5c3:728 :1] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723383] [e41b8682b5c3:728 :9] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :9:1210] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723399] [e41b8682b5c3:728 :2] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723416] [e41b8682b5c3:728 :3] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723472] [e41b8682b5c3:728 :5] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723388] [e41b8682b5c3:728 :0] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723382] [e41b8682b5c3:728 :8] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :8:1207] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723547] [e41b8682b5c3:728 :10] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723693] [e41b8682b5c3:728 :8] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723592] [e41b8682b5c3:728 :9] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723542] [e41b8682b5c3:728 :11] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :11:1221] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723744] [e41b8682b5c3:728 :11] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723744] [e41b8682b5c3:728 :12] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[1756463886.723762] [e41b8682b5c3:728 :12] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723766] [e41b8682b5c3:728 :13] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :12:1220] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723780] [e41b8682b5c3:728 :12] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :13:1252] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723787] [e41b8682b5c3:728 :13] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723784] [e41b8682b5c3:728 :14] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 4 was not set in ucs
[e41b8682b5c3:728 :14:1240] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723817] [e41b8682b5c3:728 :15] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :15:1208] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723822] [e41b8682b5c3:728 :14] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723838] [e41b8682b5c3:728 :16] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :16:728] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723838] [e41b8682b5c3:728 :15] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723861] [e41b8682b5c3:728 :17] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :17:1200] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723866] [e41b8682b5c3:728 :16] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723889] [e41b8682b5c3:728 :18] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :18:1242] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723891] [e41b8682b5c3:728 :17] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723910] [e41b8682b5c3:728 :19] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :19:1254] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723911] [e41b8682b5c3:728 :18] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723935] [e41b8682b5c3:728 :20] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 1 was not set in ucs
[e41b8682b5c3:728 :20:1239] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723960] [e41b8682b5c3:728 :20] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723961] [e41b8682b5c3:728 :21] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :21:1212] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723938] [e41b8682b5c3:728 :19] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.723984] [e41b8682b5c3:728 :22] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :22:1244] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.723989] [e41b8682b5c3:728 :21] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724012] [e41b8682b5c3:728 :23] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :23:1255] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724013] [e41b8682b5c3:728 :22] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724038] [e41b8682b5c3:728 :24] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 8 was not set in ucs
[e41b8682b5c3:728 :24:1246] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724041] [e41b8682b5c3:728 :23] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724062] [e41b8682b5c3:728 :25] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :25:1214] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724067] [e41b8682b5c3:728 :24] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724096] [e41b8682b5c3:728 :26] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :26:1218] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724098] [e41b8682b5c3:728 :25] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724121] [e41b8682b5c3:728 :27] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :27:1250] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724126] [e41b8682b5c3:728 :26] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724152] [e41b8682b5c3:728 :28] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[1756463886.724173] [e41b8682b5c3:728 :28] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :28:1217] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724155] [e41b8682b5c3:728 :27] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724179] [e41b8682b5c3:728 :29] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 4 was not set in ucs
[e41b8682b5c3:728 :29:1230] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724188] [e41b8682b5c3:728 :28] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724217] [e41b8682b5c3:728 :30] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 4 was not set in ucs
[e41b8682b5c3:728 :30:1262] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724224] [e41b8682b5c3:728 :29] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724246] [e41b8682b5c3:728 :30] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724243] [e41b8682b5c3:728 :31] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :31:1249] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724278] [e41b8682b5c3:728 :31] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724279] [e41b8682b5c3:728 :32] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :32:1202] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724301] [e41b8682b5c3:728 :33] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 4 was not set in ucs
[e41b8682b5c3:728 :33:1234] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724306] [e41b8682b5c3:728 :32] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724319] [e41b8682b5c3:728 :34] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 11 was not set in ucs
[1756463886.724321] [e41b8682b5c3:728 :33] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724342] [e41b8682b5c3:728 :35] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 11 was not set in ucs
[e41b8682b5c3:728 :35:1206] Caught signal 7 (Bus error: nonexistent physical address)
[e41b8682b5c3:728 :34:1238] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724365] [e41b8682b5c3:728 :35] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724362] [e41b8682b5c3:728 :36] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :36:1248] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724367] [e41b8682b5c3:728 :34] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724392] [e41b8682b5c3:728 :37] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[1756463886.724412] [e41b8682b5c3:728 :37] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :37:1257] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724417] [e41b8682b5c3:728 :38] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[1756463886.724393] [e41b8682b5c3:728 :36] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :38:1211] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724442] [e41b8682b5c3:728 :39] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :39:1243] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724462] [e41b8682b5c3:728 :39] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724462] [e41b8682b5c3:728 :40] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :40:1259] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724448] [e41b8682b5c3:728 :38] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724428] [e41b8682b5c3:728 :37] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724485] [e41b8682b5c3:728 :41] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 4 was not set in ucs
[e41b8682b5c3:728 :41:1227] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724490] [e41b8682b5c3:728 :40] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724525] [e41b8682b5c3:728 :41] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724521] [e41b8682b5c3:728 :42] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :42:1225] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724555] [e41b8682b5c3:728 :42] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724555] [e41b8682b5c3:728 :43] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[1756463886.724574] [e41b8682b5c3:728 :43] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724577] [e41b8682b5c3:728 :44] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :43:1245] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724590] [e41b8682b5c3:728 :44] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :44:1213] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724595] [e41b8682b5c3:728 :45] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 11 was not set in ucs
[1756463886.724615] [e41b8682b5c3:728 :45] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724618] [e41b8682b5c3:728 :46] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 1 was not set in ucs
[e41b8682b5c3:728 :45:1224] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724631] [e41b8682b5c3:728 :45] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724604] [e41b8682b5c3:728 :44] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724638] [e41b8682b5c3:728 :47] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :47:1231] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724592] [e41b8682b5c3:728 :43] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :46:1256] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724671] [e41b8682b5c3:728 :48] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 1 was not set in ucs
[e41b8682b5c3:728 :48:1203] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724676] [e41b8682b5c3:728 :47] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724703] [e41b8682b5c3:728 :49] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 11 was not set in ucs
[1756463886.724702] [e41b8682b5c3:728 :48] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :49:1204] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724726] [e41b8682b5c3:728 :50] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[1756463886.724690] [e41b8682b5c3:728 :46] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :50:1236] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724749] [e41b8682b5c3:728 :50] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724732] [e41b8682b5c3:728 :49] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724747] [e41b8682b5c3:728 :51] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :51:1226] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724780] [e41b8682b5c3:728 :52] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 4 was not set in ucs
[e41b8682b5c3:728 :52:1258] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724786] [e41b8682b5c3:728 :51] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724802] [e41b8682b5c3:728 :53] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :53:1228] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724804] [e41b8682b5c3:728 :52] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724824] [e41b8682b5c3:728 :54] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :54:1260] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724830] [e41b8682b5c3:728 :53] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724856] [e41b8682b5c3:728 :54] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724852] [e41b8682b5c3:728 :55] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 11 was not set in ucs
[e41b8682b5c3:728 :55:1232] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724887] [e41b8682b5c3:728 :55] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724885] [e41b8682b5c3:728 :56] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[e41b8682b5c3:728 :56:1222] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724909] [e41b8682b5c3:728 :56] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724910] [e41b8682b5c3:728 :57] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 7 was not set in ucs
[1756463886.724928] [e41b8682b5c3:728 :57] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[e41b8682b5c3:728 :57:1201] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724933] [e41b8682b5c3:728 :58] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 11 was not set in ucs
[e41b8682b5c3:728 :58:1205] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724941] [e41b8682b5c3:728 :57] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724955] [e41b8682b5c3:728 :59] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 4 was not set in ucs
[e41b8682b5c3:728 :59:1237] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724961] [e41b8682b5c3:728 :58] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.724984] [e41b8682b5c3:728 :60] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 4 was not set in ucs
[e41b8682b5c3:728 :60:1247] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.724985] [e41b8682b5c3:728 :59] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.725007] [e41b8682b5c3:728 :61] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 4 was not set in ucs
[e41b8682b5c3:728 :61:1223] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.725011] [e41b8682b5c3:728 :60] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.725039] [e41b8682b5c3:728 :61] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.725037] [e41b8682b5c3:728 :62] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 11 was not set in ucs
[e41b8682b5c3:728 :62:1233] Caught signal 7 (Bus error: nonexistent physical address)
[1756463886.725071] [e41b8682b5c3:728 :62] spinlock.c:29 UCX WARN ucs_recursive_spinlock_destroy() failed: busy
[1756463886.725070] [e41b8682b5c3:728 :63] debug.c:1300 UCX WARN ucs_debug_disable_signal: signal 4 was not set in ucs
[e41b8682b5c3:728 :63:1215] Caught signal 7 (Bus error: nonexistent physical address)
==== backtrace (tid: 1261) ====
0 /opt/hpcx/ucx/lib/libucs.so.0(ucs_handle_error+0x2e4) [0x7f60de994774]
1 /opt/hpcx/ucx/lib/libucs.so.0(+0x3796a) [0x7f60de99496a]
2 /opt/hpcx/ucx/lib/libucs.so.0(+0x37bdf) [0x7f60de994bdf]
3 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x45330) [0x7f614ebe6330]
4 /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so(+0x67e105b) [0x7f613b6a005b]
5 /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so(_ZNK2at18TensorIteratorBase15serial_for_eachEN3c1012function_refIFvPPcPKlllEEENS_5RangeE+0x4e2) [0x7f6136304a52]
6 /usr/local/lib/python3.12/dist-packages/torch/lib/libtorch_cpu.so(+0x1445ba3) [0x7f6136304ba3]
7 /usr/lib/x86_64-linux-gnu/libgomp.so.1(+0x2279f) [0x7f614e2a179f]
8 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x7f614ec3daa4]
9 /usr/lib/x86_64-linux-gnu/libc.so.6(__clone+0x44) [0x7f614eccaa34]
=================================
trtllm_serve_start.sh: line 3: 728 Bus error (core dumped) CUDA_VISIBLE_DEVICES=7 trtllm-serve /data1/llm-models/qwen/Qwen2___5-VL-7B-Instruct --extra_llm_api_options ./extra-llm-api-config.yml --max_batch_size 10 --max_num_tokens 40960 --max_seq_len 8192 --trust_remote_code
additional notes
The issue does not occur in TensorRT-LLM v1.0.0rc6
Before submitting a new issue...
- Make sure you already searched for relevant issues, and checked the documentation and examples for answers to frequently asked questions.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.