modelscope / modelscope/ms-swift

vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.

Open
#7,280 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

Describe the bug
使用
IMAGE_MAX_TOKEN_NUM=128 CUDA_VISIBLE_DEVICES=0,1,2,3 swift deploy --model /Exp008/v0-20251231-163417/iter_0000600_hf --infer_backend vllm --port 8001 --api_key abc123 --served_model_name Exp008_step600 --vllm_tensor_parallel_size 4 --max_new_tokens 30000 --vllm_gpu_memory_utilization 0.8部署Qwen3-vl-32B-think进行推理,16并发推理,会遇到以下问题

[INFO:swift] Traceback (most recent call last): File "/root/training_env_sft/ms-swift/swift/llm/infer/deploy.py", line 198, in create_chat_completion res_or_gen = await self.infer_async(infer_request, request_config, template=self.template, **infer_kwargs) File "/root/training_env_sft/ms-swift/swift/llm/infer/infer_engine/vllm_engine.py", line 806, in infer_async return await self._infer_full_async(**kwargs) File "/root/training_env_sft/ms-swift/swift/llm/infer/infer_engine/vllm_engine.py", line 671, in _infer_full_async async for result in result_generator: File "/root/training_env_sft/.venv/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 370, in generate q = await self.add_request( File "/root/training_env_sft/.venv/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 276, in add_request raise EngineDeadError() vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.

INFO: 28.18.40.30:47747 - "POST /v1/chat/completions HTTP/1.1" 400 Bad Request

Your hardware and system info
这是我的环境信息:
vllm 0.11.0
torch 2.8.0
ms-swift 3.13.0.dev0

Additional context
期待您的回复

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with swift/llm/infer/deploy.py at create_chat_completion and swift/llm/infer/infer_engine/vllm_engine.py at infer_async and _infer_full_async. Reproduce the deployment command with the reported environment and inspect the missing EngineCore stack trace, including vllm/v1/engine/async_llm.py. Done means identifying the root cause of EngineDeadError under 16 concurrent requests and documenting or validating the fix.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.