modelscope / modelscope/ms-swift

910b上infer一直报错,ERR00007 PTA feature not supported

Open
#6,684 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

npu stale
Dominant language
Python
Stars
15.7k
Forks
1.7k
Avg merge
1d 16h
Merged PRs (30d)
136

Description

问题命令:
swift infer \ --adapters /cache/model/qwen8b-lora/sft-1120-1/v0-20251120-152355/checkpoint-80 \ --stream true \ --infer_backend vllm \ --vllm_max_model_len 8192 \ --temperature 0 \ --max_new_tokens 2048

报错:
RuntimeError: Cannot run aclop operators during NPU graph capture. Current working aclop is MatMul. If you need this call to be captured, please try to set torch.npu.config.allow_internal_format = False. If still fail, the operator needs aclnn implementation and please file an issue. Current npuStreamCaptureStatus: npuStreamCaptureStatusActive

具体报错:
npuStreamCaptureStatusActive
(EngineCore_DP0 pid=2011231) ERROR 11-20 20:48:03 [core.py:708] [ERROR] 2025-11-20-20:48:03 (PID:2011231, Device:0, RankID:-1) ERR00007 PTA feature not supported
(EngineCore_DP0 pid=2011231) Process EngineCore_DP0:
(EngineCore_DP0 pid=2011231) Traceback (most recent call last):
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/multiprocessing/process.py", line 314, in bootstrap
(EngineCore_DP0 pid=2011231) self.run()
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/multiprocessing/process.py", line 108, in run
(EngineCore_DP0 pid=2011231) self.target(*self.args, **self.kwargs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 712, in run_engine_core
(EngineCore_DP0 pid=2011231) raise e
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 699, in run_engine_core
(EngineCore_DP0 pid=2011231) engine_core = EngineCoreProc(*args, **kwargs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 498, in init
(EngineCore_DP0 pid=2011231) super().init(vllm_config, executor_class, log_stats,
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 92, in init
(EngineCore_DP0 pid=2011231) self.initialize_kv_caches(vllm_config)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 207, in initialize_kv_caches
(EngineCore_DP0 pid=2011231) self.model_executor.initialize_from_config(kv_cache_configs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 75, in initialize_from_config
(EngineCore_DP0 pid=2011231) self.collective_rpc("compile_or_warm_up_model")
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/executor/uniproc_executor.py", line 83, in collective_rpc
(EngineCore_DP0 pid=2011231) return [run_method(self.driver_worker, method, args, kwargs)]
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/utils/init.py", line 3122, in run_method
(EngineCore_DP0 pid=2011231) return func(*args, **kwargs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm_ascend/worker/worker_v1.py", line 329, in compile_or_warm_up_model
(EngineCore_DP0 pid=2011231) self.model_runner.capture_model()
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm_ascend/worker/model_runner_v1.py", line 3561, in capture_model
(EngineCore_DP0 pid=2011231) self.capture_model()
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm_ascend/worker/model_runner_v1.py", line 3499, in capture_model
(EngineCore_DP0 pid=2011231) self.capture_aclgraphs(
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm_ascend/worker/model_runner_v1.py", line 3473, in capture_aclgraphs
(EngineCore_DP0 pid=2011231) self.dummy_run(num_tokens,
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/torch/utils/contextlib.py", line 116, in decorate_context
(EngineCore_DP0 pid=2011231) return func(*args, **kwargs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm_ascend/worker/model_runner_v1.py", line 2494, in dummy_run
(EngineCore_DP0 pid=2011231) hidden_states = self.generate_dummy_run_hidden_states(
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm_ascend/worker/model_runner_v1.py", line 2315, in generate_dummy_run_hidden_states
(EngineCore_DP0 pid=2011231) hidden_states = self.model(input_ids=input_ids,
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in wrapped_call_impl
(EngineCore_DP0 pid=2011231) return self.call_impl(*args, **kwargs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in call_impl
(EngineCore_DP0 pid=2011231) return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/model_executor/models/qwen3.py", line 323, in forward
(EngineCore_DP0 pid=2011231) hidden_states = self.model(input_ids, positions, intermediate_tensors,
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 317, in call
(EngineCore_DP0 pid=2011231) model_output = self.forward(*args, **kwargs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/model_executor/models/qwen2.py", line 341, in forward
(EngineCore_DP0 pid=2011231) def forward(
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in wrapped_call_impl
(EngineCore_DP0 pid=2011231) return self.call_impl(*args, **kwargs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in call_impl
(EngineCore_DP0 pid=2011231) return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/torch/dynamo/eval_frame.py", line 838, in fn
(EngineCore_DP0 pid=2011231) return fn(*args, **kwargs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/torch/fx/graph_module.py", line 830, in call_wrapped
(EngineCore_DP0 pid=2011231) return self.wrapped_call(self, *args, **kwargs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/torch/fx/graph_module.py", line 406, in call
(EngineCore_DP0 pid=2011231) raise e
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/torch/fx/graph_module.py", line 393, in call
(EngineCore_DP0 pid=2011231) return super(self.cls, obj).call(*args, **kwargs) # type: ignore[misc]
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1751, in wrapped_call_impl
(EngineCore_DP0 pid=2011231) return self.call_impl(*args, **kwargs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1762, in call_impl
(EngineCore_DP0 pid=2011231) return forward_call(*args, **kwargs)
(EngineCore_DP0 pid=2011231) File "<eval_with_key>.74", line 804, in forward
(EngineCore_DP0 pid=2011231) submod_0 = self.submod_0(l_input_ids
, s0, l_self_modules_embed_tokens_parameters_weight
, l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight
, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_modules_base_layer_parameters_weight
, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_punica_wrapper_seq_lengths, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_punica_wrapper_lora_indices_per_batch, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_lora_a_stacked_0
, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_lora_a_stacked_1
, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_lora_a_stacked_2
, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_lora_b_stacked_0
, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_lora_b_stacked_1
, l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_lora_b_stacked_2
, l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight
, l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight
, l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache
, l_positions
, s1); l_input_ids
= l_self_modules_embed_tokens_parameters_weight
= l_self_modules_layers_modules_0_modules_input_layernorm_parameters_weight
= l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_modules_base_layer_parameters_weight
= l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_lora_a_stacked_0
= l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_lora_a_stacked_1
= l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_lora_a_stacked_2
= l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_lora_b_stacked_0
= l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_lora_b_stacked_1
= l_self_modules_layers_modules_0_modules_self_attn_modules_qkv_proj_lora_b_stacked_2
= l_self_modules_layers_modules_0_modules_self_attn_modules_q_norm_parameters_weight
= l_self_modules_layers_modules_0_modules_self_attn_modules_k_norm_parameters_weight
= l_self_modules_layers_modules_0_modules_self_attn_modules_rotary_emb_buffers_cos_sin_cache
= l_positions_ = None
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm_ascend/compilation/acl_graph.py", line 155, in call
(EngineCore_DP0 pid=2011231) output = self.runnable(*args, **kwargs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/compilation/cuda_piecewise_backend.py", line 96, in call
(EngineCore_DP0 pid=2011231) return self.compiled_graph_for_general_shape(*args)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/torch/fx/graph_module.py", line 830, in call_wrapped
(EngineCore_DP0 pid=2011231) return self._wrapped_call(self, *args, **kwargs)
(EngineCore_DP0 pid=2011231) File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/torch/fx/graph_module.py", line 404, in call
(EngineCore_DP0 pid=2011231) raise e.with_traceback(None) # noqa: B904
(EngineCore_DP0 pid=2011231) RuntimeError: Cannot run aclop operators during NPU graph capture. Current working aclop is MatMul. If you need this call to be captured, please try to set torch.npu.config.allow_internal_format = False. If still fail, the operator needs aclnn implementation and please file an issue. Current npuStreamCaptureStatus: npuStreamCaptureStatusActive
(EngineCore_DP0 pid=2011231) [ERROR] 2025-11-20-20:48:03 (PID:2011231, Device:0, RankID:-1) ERR00007 PTA feature not supported
Traceback (most recent call last):
File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/swift/cli/infer.py", line 5, in
infer_main()
File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/swift/llm/infer/infer.py", line 301, in infer_main
return SwiftInfer(args).main()
File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/swift/llm/infer/infer.py", line 39, in init
self.infer_engine = self.get_infer_engine(args, self.template)
File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/swift/llm/infer/infer.py", line 86, in get_infer_engine
return infer_engine_cls(**kwargs)
File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/swift/llm/infer/infer_engine/vllm_engine.py", line 140, in init
self._prepare_engine()
File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/swift/llm/infer/infer_engine/vllm_engine.py", line 150, in _prepare_engine
engine = llm_engine_cls.from_engine_args(self.engine_args)
File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/v1/engine/llm_engine.py", line 177, in from_engine_args
return cls(vllm_config=vllm_config,
File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/v1/engine/llm_engine.py", line 114, in init
self.engine_core = EngineCoreClient.make_client(
File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 80, in make_client
return SyncMPClient(vllm_config, executor_class, log_stats)
File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 602, in init
super().init(
File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 448, in init
with launch_core_engines(vllm_config, executor_class,
File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/contextlib.py", line 142, in exit
next(self.gen)
File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 732, in launch_core_engines
wait_for_engine_startup(
File "/home/ma-user/anaconda3/envs/vllm0.11.0/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 785, in wait_for_engine_startup
raise RuntimeError("Engine core initialization failed. "
RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
[ERROR] 2025-11-20-20:48:14 (PID:2011056, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exceptio`

Your hardware and system info
CANN版本8.2
vllm版本0.11.0
torch-npu版本2.7.1
ms_swift版本3.10.1

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the supplied swift infer command and begin with the traceback locations in vllm_ascend/worker/model_runner_v1.py, especially capture_model, _capture_aclgraphs, and _dummy_run. Compare the NPU graph-capture path with the Qwen3 call in vllm/model_executor/models/qwen3.py; done means inference no longer fails with ERR00007 PTA feature not supported.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.