disaggregated_serving deployment error

System Info

• GPU H20*8 use gpu0, 1
• model: QWQ-32B

Who can help?

No response

Information

• The official example scripts
• My own modified scripts

Tasks

• An officially supported task in the examples folder (such as GLUE/SQuAD, ...)
• My own task or dataset (give details below)

Reproduction

• Full reference documentation: https://github.com/NVIDIA/TensorRT-LLM/tree/v0.20.0/triton_backend/all_models/disaggregated_serving

• python3 /opt/app/TensorRT-LLM/triton_backend/scripts/launch_triton_server.py
  --world_size 3
  --tensorrt_llm_model_name context,generation
  --multi-model --disable-spawn-processes
  --model_repo=/opt/app/disaggregated_serving/

• python3 inflight_batcher_llm/client/end_to_end_grpc_client.py -S -p "Machine learning is"

Received an error from server: in ensemble 'ensemble', Failed to process the request(s) for model 'disaggregated_serving_bls_0_0', message: TritonModelException: Context model context failed with error: [TensorRT-LLM][ERROR] Assertion failed: Shutdown called, cannot enqueue requests (/workspace/tensorrt_llm/cpp/tensorrt_llm/executor/executorImpl.cpp:870) 1 0x7f1907ce1341 tensorrt_llm::common::throwRuntimeError(char const*, int, char const*) + 97 2 0x7f1908e0e0ed tensorrt_llm::executor::Executor::Impl::enqueueRequests(tensorrt_llm::common::ArrayView<tensorrt_llm::executor::Request const> const&) + 1853 3 0x7f1908e0e299 tensorrt_llm::executor::Executor::Impl::enqueueRequests(std::vector<tensorrt_llm::executor::Request, std::allocator<tensorrt_llm::executor::Request> > const&) + 57 4 0x7f1908dff374 tensorrt_llm::executor::Executor::enqueueRequests(std::vector<tensorrt_llm::executor::Request, std::allocator<tensorrt_llm::executor::Request> > const&) + 36 5 0x7f19e64f2f3d triton::backend::inflight_batcher_llm::ModelInstanceState::enqueue(TRITONBACKEND_Request**, unsigned int) + 1261 6 0x7f19e64e76fc TRITONBACKEND_ModelInstanceExecute + 92 7 0x7f1bfc5d0507 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x19b507) [0x7f1bfc5d0507] 8 0x7f1bfc5d0bfb /opt/tritonserver/bin/../lib/libtritonserver.so(+0x19bbfb) [0x7f1bfc5d0bfb] 9 0x7f1bfc6f34f5 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x2be4f5) [0x7f1bfc6f34f5] 10 0x7f1bfc5d4a23 /opt/tritonserver/bin/../lib/libtritonserver.so(+0x19fa23) [0x7f1bfc5d4a23] 11 0x7f1bfbd90db4 /usr/lib/x86_64-linux-gnu/libstdc++.so.6(+0xecdb4) [0x7f1bfbd90db4] 12 0x7f1bfba9caa4 /usr/lib/x86_64-linux-gnu/libc.so.6(+0x9caa4) [0x7f1bfba9caa4] 13 0x7f1bfbb29c3c /usr/lib/x86_64-linux-gnu/libc.so.6(+0x129c3c) [0x7f1bfbb29c3c]

Expected behavior

no error

actual behavior

error

additional notes

none