[Feature] ascend 310p上执行图模式

Open

#3,119 opened on Feb 7, 2025 9 comments 0 reactions 2 assignees Claimed by @yao-fengchen View on GitHub

Dominant language: Python
Stars: 8.1k
Forks: 748
Avg merge: 6d 2h
Merged PRs (30d): 54

Description

### Motivation

在ascend 310p上创建以下环境torch=2.3.1+cpu、torch-npu=2.3.1.post4

LMdeploy和dlinfer代码分别使用[DeepLink-org:support_310P](https://github.com/DeepLink-org/lmdeploy/tree/support_310P)和[yao-fengchen:support_310P](https://github.com/yao-fengchen/dlinfer/tree/support_310P)，

ascend 310p上进行图模式的推理任务时

```python
from lmdeploy import pipeline
from lmdeploy import PytorchEngineConfig, GenerationConfig

if __name__ == "__main__":
pipe = pipeline("/mnt/data/llm/Qwen1.5-7B-Chat/",
backend_config=PytorchEngineConfig(
tp=1,
device_type="ascend",
dtype='float16',
eager_mode=False,
cache_max_entry_count=0.5))
# question = ["Shanghai is"]
question = ["Shanghai is the largest city in China. Please introduce it."]
response = pipe(question, gen_config=GenerationConfig(max_new_tokens=10))
print(response)

```

存在以下问题

```python
/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch_npu/utils/collect_env.py:59: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/latest owner does not match the current owner.
warnings.warn(f"Warning: The {path} owner does not match the current owner.")
/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch_npu/utils/collect_env.py:59: UserWarning: Warning: The /usr/local/Ascend/ascend-toolkit/8.0.0.alpha001/x86_64-linux/ascend_toolkit_install.info owner does not match the current owner.
warnings.warn(f"Warning: The {path} owner does not match the current owner.")
/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch_npu/contrib/transfer_to_npu.py:292: ImportWarning:
*************************************************************************************************************
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.Generator, torch.set_default_device, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.nn.Module.to, torch.nn.Module.to_empty
*************************************************************************************************************

warnings.warn(msg, ImportWarning)
/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch_npu/contrib/transfer_to_npu.py:247: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
warnings.warn(msg, RuntimeWarning)
2025-02-07 22:25:43,668 - lmdeploy - WARNING - transformers.py:22 - LMDeploy requires transformers version: [4.33.0 ~ 4.46.1], but found version: 4.48.0
Loading weights from safetensors: 100%|█████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:01<00:00, 2.21it/s]
/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/backends/dlinfer/ascend/graph_runner.py:51: RuntimeWarning:

************************************************************
Graph mode is an experimental feature. We currently
support both dense and Mixture of Experts (MoE) models
with bf16 and fp16 data types.
If graph mode does not function correctly with your model,
please consider using eager mode as an alternative.
************************************************************

warnings.warn(
2025-02-07 22:26:04,020 - lmdeploy - WARNING - async_engine.py:625 - GenerationConfig: GenerationConfig(n=1, max_new_tokens=50, do_sample=False, top_p=1.0, top_k=50, min_p=0.0, temperature=0.8, repetition_penalty=1.0, ignore_eos=False, random_seed=None, stop_words=None, bad_words=None, stop_token_ids=[151645], bad_token_ids=None, min_new_tokens=None, skip_special_tokens=True, spaces_between_special_tokens=True, logprobs=None, response_format=None, logits_processors=None, output_logits=None, output_last_hidden_state=None)
2025-02-07 22:26:04,020 - lmdeploy - WARNING - async_engine.py:626 - Since v0.6.0, lmdeploy add `do_sample` in GenerationConfig. It defaults to False, meaning greedy decoding. Please set `do_sample=True` if sampling decoding is needed
/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch_npu/utils/storage.py:38: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
if self.device.type != 'cpu':
2025-02-07 22:26:12,933 - lmdeploy - ERROR - engine.py:907 - Task failed
Traceback (most recent call last):
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/engine.py", line 902, in __task_callback
task.result()
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/engine.py", line 860, in _async_loop_background
await self._async_step_background(
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/engine.py", line 729, in _async_step_background
output = await self._async_model_forward(inputs,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/utils.py", line 234, in __tmp
return (await func(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/engine.py", line 627, in _async_model_forward
ret = await __forward(inputs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/engine.py", line 604, in __forward
return await self.model_agent.async_forward(inputs, swap_in_map=swap_in_map, swap_out_map=swap_out_map)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/model_agent.py", line 256, in async_forward
output = self._forward_impl(inputs, swap_in_map=swap_in_map, swap_out_map=swap_out_map)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/model_agent.py", line 239, in _forward_impl
output = model_forward(
^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/model_agent.py", line 151, in model_forward
output = model(**input_dict)
^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/backends/graph_runner.py", line 24, in __call__
return self.model(**kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 451, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 921, in catch_errors
return callback(frame, cache_entry, hooks, frame_state, skip=1)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 400, in _convert_frame_assert
return _compile(
^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 676, in _compile
guarded_code = compile_inner(code, one_graph, hooks, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 535, in compile_inner
out_code = transform_code_object(code, transform)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/bytecode_transformation.py", line 1036, in transform_code_object
transformations(instructions, code_options)
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 165, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/convert_frame.py", line 500, in transform
tracer.run()
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2149, in run
super().run()
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 810, in run
and self.step()
^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 773, in step
getattr(self, inst.opname)(inst)
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/symbolic_convert.py", line 2268, in RETURN_VALUE
self.output.compile_subgraph(
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 981, in compile_subgraph
self.compile_and_call_fx_graph(tx, list(reversed(stack_values)), root)
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1178, in compile_and_call_fx_graph
compiled_fn = self.call_user_compiler(gm)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1251, in call_user_compiler
raise BackendCompilerFailed(self.compiler_fn, e).with_traceback(
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/output_graph.py", line 1232, in call_user_compiler
compiled_fn = compiler_fn(gm, self.example_inputs())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
compiled_gm = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/repro/after_dynamo.py", line 117, in debug_wrapper
compiled_gm = compiler_fn(gm, example_inputs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/__init__.py", line 1770, in __call__
return self.compiler_fn(model_, inputs_, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/dlinfer_support_310P/dlinfer/graph/dicp/vendor/AtbGraph/__init__.py", line 7, in atbgraph
return compile_fx(gm, fake_input_tensor, "atbgraph")
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/dlinfer_support_310P/dlinfer/graph/dicp/dynamo_bridge/compile_fx.py", line 103, in compile_fx
return compile_fx_210(model_, example_inputs_, backend, inner_compile)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/dlinfer_support_310P/dlinfer/graph/dicp/dynamo_bridge/compile_fx.py", line 255, in compile_fx_210
return aot_autograd(
^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/backends/common.py", line 58, in compiler_fn
cg = aot_module_simplified(gm, example_inputs, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 903, in aot_module_simplified
compiled_fn = create_aot_dispatcher_function(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 628, in create_aot_dispatcher_function
compiled_fn = compiler_fn(flat_fn, fake_flat_args, aot_config, fw_metadata=fw_metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 443, in aot_wrapper_dedupe
return compiler_fn(flat_fn, leaf_flat_args, aot_config, fw_metadata=fw_metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 648, in aot_wrapper_synthetic_base
return compiler_fn(flat_fn, flat_args, aot_config, fw_metadata=fw_metadata)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 119, in aot_dispatch_base
compiled_fw = compiler(fw_module, updated_flat_args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/utils.py", line 262, in time_wrapper
r = func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/dlinfer_support_310P/dlinfer/graph/dicp/dynamo_bridge/compile_fx.py", line 217, in fw_compiler_base
return inner_compile(
^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/yzk/dlinfer_support_310P/dlinfer/graph/dicp/dynamo_bridge/compile_fx.py", line 81, in compile_fx_inner
gt = GraphTransformer(gm, backend)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/dlinfer_support_310P/dlinfer/graph/dicp/dynamo_bridge/graph.py", line 53, in __init__
from dlinfer.graph.dicp.vendor.AtbGraph.codegen.atb import AtbCodegen
File "/home/yzk/dlinfer_support_310P/dlinfer/graph/dicp/vendor/AtbGraph/codegen/atb.py", line 8, in
from dlinfer.graph.dicp.vendor.AtbGraph.codegen.atb_graph import Graph, parse_graph
File "/home/yzk/dlinfer_support_310P/dlinfer/graph/dicp/vendor/AtbGraph/codegen/atb_graph.py", line 7, in
from dlinfer.graph.dicp.vendor.AtbGraph.codegen import atb_infer_param as infer_param
File "/home/yzk/dlinfer_support_310P/dlinfer/graph/dicp/vendor/AtbGraph/codegen/atb_infer_param.py", line 138, in
@dataclass
^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/dataclasses.py", line 1232, in dataclass
return wrap(cls)
^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/dataclasses.py", line 1222, in wrap
return _process_class(cls, init, repr, eq, order, unsafe_hash,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/dataclasses.py", line 958, in _process_class
cls_fields.append(_get_field(cls, name, type, kw_only))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/dataclasses.py", line 815, in _get_field
raise ValueError(f'mutable default {type(f.default)} for field '
torch._dynamo.exc.BackendCompilerFailed: backend='atbgraph' raised:
ValueError: mutable default for field normParam is not allowed: use default_factory

Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information

You can suppress this exception and fall back to eager by setting:
import torch._dynamo
torch._dynamo.config.suppress_errors = True

2025-02-07 22:26:12,937 - lmdeploy - ERROR - async_engine.py:773 - session 0 finished, reason "error"
2025-02-07 22:26:12,937 - lmdeploy - ERROR - async_engine.py:773 - session 1 finished, reason "error"
2025-02-07 22:26:12,938 - lmdeploy - ERROR - async_engine.py:773 - session 2 finished, reason "error"
[Response(text='internal error happened', generate_token_len=0, input_token_len=22, finish_reason='error', token_ids=[], logprobs=None, logits=None, last_hidden_state=None, index=0), Response(text='internal error happened', generate_token_len=0, input_token_len=22, finish_reason='error', token_ids=[], logprobs=None, logits=None, last_hidden_state=None, index=1), Response(text='internal error happened', generate_token_len=0, input_token_len=23, finish_reason='error', token_ids=[], logprobs=None, logits=None, last_hidden_state=None, index=2)]

```

在定义数据类时，某个字段使用了可变对象作为默认值，而Python的dataclass不允许这样做。根据Python的dataclass规范，如果字段的默认值是可变对象（比如列表、字典或自定义类的实例），应该使用`default_factory`来生成默认值，而不是直接赋值。修改对应代码为下图所示，并执行source /usr/local/Ascend/nnal/atb/set_env.sh

![Image](https://github.com/user-attachments/assets/9a36c346-65fb-4196-9c78-301eab954d09)

出现了如下错误

warnings.warn(msg, ImportWarning)
/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch_npu/contrib/transfer_to_npu.py:247: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
warnings.warn(msg, RuntimeWarning)
2025-02-07 23:04:46,154 - lmdeploy - WARNING - transformers.py:22 - LMDeploy requires transformers version: [4.33.0 ~ 4.46.1], but found version: 4.48.0
Loading weights from safetensors: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:01<00:00, 2.28it/s]
/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/backends/dlinfer/ascend/graph_runner.py:51: RuntimeWarning:

warnings.warn(
2025-02-07 23:05:06,454 - lmdeploy - WARNING - async_engine.py:625 - GenerationConfig: GenerationConfig(n=1, max_new_tokens=50, do_sample=False, top_p=1.0, top_k=50, min_p=0.0, temperature=0.8, repetition_penalty=1.0, ignore_eos=False, random_seed=None, stop_words=None, bad_words=None, stop_token_ids=[151645], bad_token_ids=None, min_new_tokens=None, skip_special_tokens=True, spaces_between_special_tokens=True, logprobs=None, response_format=None, logits_processors=None, output_logits=None, output_last_hidden_state=None)
2025-02-07 23:05:06,454 - lmdeploy - WARNING - async_engine.py:626 - Since v0.6.0, lmdeploy add `do_sample` in GenerationConfig. It defaults to False, meaning greedy decoding. Please set `do_sample=True` if sampling decoding is needed
/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch_npu/utils/storage.py:38: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly. To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
if self.device.type != 'cpu':
mki_log mkdir /home/yzk/atb/
mki_log mkdir /home/yzk/atb/log
[2025-02-07 23:05:28.176] [dicp] [error] [model.cpp:266] execute node[0] fail, error code: 3
[2025-02-07 23:05:28.176] [dicp] [critical] [model.cpp:241] 0 execute node[0] failed, error code: 3
2025-02-07 23:05:28,179 - lmdeploy - ERROR - engine.py:907 - Task failed
Traceback (most recent call last):
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/engine.py", line 902, in __task_callback
task.result()
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/engine.py", line 860, in _async_loop_background
await self._async_step_background(
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/engine.py", line 729, in _async_step_background
output = await self._async_model_forward(inputs,
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/utils.py", line 234, in __tmp
return (await func(*args, **kwargs))
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/engine.py", line 627, in _async_model_forward
ret = await __forward(inputs)
^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/engine.py", line 604, in __forward
return await self.model_agent.async_forward(inputs, swap_in_map=swap_in_map, swap_out_map=swap_out_map)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/model_agent.py", line 256, in async_forward
output = self._forward_impl(inputs, swap_in_map=swap_in_map, swap_out_map=swap_out_map)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/model_agent.py", line 239, in _forward_impl
output = model_forward(
^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/engine/model_agent.py", line 151, in model_forward
output = model(**input_dict)
^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/backends/graph_runner.py", line 24, in __call__
return self.model(**kwargs)
^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 451, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/lmdeploy_support_310P/lmdeploy/pytorch/models/qwen2.py", line 314, in forward
def forward(
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/eval_frame.py", line 451, in _fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_dynamo/external_utils.py", line 36, in inner
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_functorch/aot_autograd.py", line 917, in forward
return compiled_fn(full_args)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/utils.py", line 89, in g
return f(*args)
^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 106, in runtime_wrapper
all_outs = call_func_at_runtime_with_args(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/utils.py", line 113, in call_func_at_runtime_with_args
out = normalize_as_list(f(args))
^^^^^^^
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/site-packages/torch/_functorch/_aot_autograd/jit_compile_runtime_wrappers.py", line 152, in rng_functionalization_wrapper
return compiled_fw(args)
^^^^^^^^^^^^^^^^^
File "/tmp/torchinductor_yzk/y2/cy2jeobsbovrkxb6lf3hmxh6e577xp33j7yz2w4boxravr64yuq6.py", line 78, in call
kernel_cpp_0(inputs, outputs, param)
File "/home/yzk/.conda/envs/yzk-lmdeploy/lib/python3.11/contextlib.py", line 81, in inner
return func(*args, **kwds)
^^^^^^^^^^^^^^^^^^^
File "/home/yzk/dlinfer_support_310P/dlinfer/graph/dicp/vendor/AtbGraph/codegen/load_and_run.py", line 17, in run
self.model.execute_out(inputs, outputs, param)
RuntimeError: Fatal error occurred: 0 execute node[0] failed, error code: 3
2025-02-07 23:05:28,182 - lmdeploy - ERROR - async_engine.py:773 - session 0 finished, reason "error"
2025-02-07 23:05:28,182 - lmdeploy - ERROR - async_engine.py:773 - session 1 finished, reason "error"
2025-02-07 23:05:28,182 - lmdeploy - ERROR - async_engine.py:773 - session 2 finished, reason "error"
[Response(text='internal error happened', generate_token_len=0, input_token_len=22, finish_reason='error', token_ids=[], logprobs=None, logits=None, last_hidden_state=None, index=0), Response(text='internal error happened', generate_token_len=0, input_token_len=22, finish_reason='error', token_ids=[], logprobs=None, logits=None, last_hidden_state=None, index=1), Response(text='internal error happened', generate_token_len=0, input_token_len=23, finish_reason='error', token_ids=[], logprobs=None, logits=None, last_hidden_state=None, index=2)]

```

这个errcode:3这个报错和[issues2745](https://github.com/InternLM/lmdeploy/issues/2745)中最后的报错一样。**请问310P目前是否具有支持图模式的计划**

### Related resources

_No response_

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

[Feature] ascend 310p上执行图模式

Description

Contributor guide

Assessment

Get new issues in your inbox