alibaba / alibaba/MNN

已获得支持mtk npu 的最新仓库mnn3.6.0,使用neuropilot-sdk-basic-7.0.8-build20240807进行天玑9300/8300 llmexport.py导出后,使用neruonsdk 对第一层prompt编译报错

Open
#4,792 2 comments 0 reactions 1 assignee Claimed by @Qxinyu View on GitHub
module:llm platform:android type:bug
Dominant language
C++
Stars
16.1k
Forks
2.4k
Avg merge
2d 23h
Merged PRs (30d)
48

Description

https://mnn-docs.readthedocs.io/en/latest/transformers/llm.html#mtk-llm
step follow above link
rope op not support
```text
sin/cos
cast(int32 -> fp16)
mul
unsupport op
```
error occur because io function _nupSupport
```python

static bool _npuSupportOp(const Op* op) {
if (gMaxKVSize > 0) {
return true;
}
if (op->type() == OpType_Attention && attn->kv_cache())
return false;
if (op->type() == OpType_LinearAttention)
return false;
return true;
}
```
prefill -> dynamic rope
decode -> static rope

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.