已获得支持mtk npu 的最新仓库mnn3.6.0,使用neuropilot-sdk-basic-7.0.8-build20240807进行天玑9300/8300 llmexport.py导出后,使用neruonsdk 对第一层prompt编译报错
Open
module:llm
platform:android
type:bug
- Dominant language
- C++
- Stars
- 16.1k
- Forks
- 2.4k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 48
Description
https://mnn-docs.readthedocs.io/en/latest/transformers/llm.html#mtk-llm
step follow above link
rope op not support
```text
sin/cos
cast(int32 -> fp16)
mul
unsupport op
```
error occur because io function _nupSupport
```python
static bool _npuSupportOp(const Op* op) {
if (gMaxKVSize > 0) {
return true;
}
if (op->type() == OpType_Attention && attn->kv_cache())
return false;
if (op->type() == OpType_LinearAttention)
return false;
return true;
}
```
prefill -> dynamic rope
decode -> static rope
Contributor guide
Assessment
This issue has not been assessed yet.