kvcache-ai / kvcache-ai/ktransformers
【ERROR】Install Ktransformers for AscendNPU
- Dominant language
- Python
- Stars
- 19.5k
- Forks
- 1.6k
- Avg merge
- 19h 32m
- Merged PRs (30d)
- 27
Description
### Reminder
- [x] I have read the above rules and searched the existing issues.
### System Info
我参考如下链接进行安装：
[ktransformers/doc/zh/DeepseekR1_V3_tutorial_zh_for_Ascend_NPU.md at main · kvcache-ai/ktransformers](https://github.com/kvcache-ai/ktransformers/blob/main/doc/zh/DeepseekR1_V3_tutorial_zh_for_Ascend_NPU.md#ktransformers%E9%83%A8%E7%BD%B2)](https://github.com/kvcache-ai/ktransformers/blob/main/doc/zh/DeepseekR1_V3_tutorial_zh_for_Ascend_NPU.md#ktransformers部署)
我的系统信息是：
- 服务器型号：Atlas A2 A800T服务器
- NPU： A2 A800T (910B3)
- CPU: HUAWEI Kunpeng 920 5250
- 内存: DDR5服务器内存（1TB）
HDK: 24.1.rc3
CANN: CANN 8.3.RC1.alpha003，配套ToolKit，Kernel和NNAL均已安装好
OS: openEuler 22.03 LTS
操作系统内核为：5.10.0-60.18.0.50.oe2203.aarch64
GCC: 10.3.1
按照教程链接顺利安装好了conda环境和python部署，并安装好了指定版本torch_npu 2.5.1
### Reproduction
将项目文件部署到机器上：
- 初始化third_party。由于此过程耗时较多，且容易受网络影响导致仓库克隆失败，建议初始化一次后，将相关文件进行打包，以便后续直接解压使用。
```bash
# 拉取最新版本
git clone https://github.com/kvcache-ai/ktransformers.git
cd archive
git submodule update --init --recursive
```
发现没有成功初始化，根据 `archive/.gitmodules` 文件内容，**手动**将这些子模块添加到主仓库中
```bash
# 在主项目根目录 (ktransformers/) 下执行以下所有命令
# 1. 添加 archive/ 下的第三方库
git submodule add https://github.com/ggerganov/llama.cpp.git archive/third_party/llama.cpp
git submodule add https://github.com/pybind/pybind11.git archive/third_party/pybind11
git submodule add https://github.com/gabime/spdlog.git archive/third_party/spdlog
git submodule add https://github.com/kvcache-ai/custom_flashinfer.git archive/third_party/custom_flashinfer
git submodule add https://github.com/Cyan4973/xxHash.git archive/third_party/xxHash
git submodule add https://github.com/jupp0r/prometheus-cpp archive/third_party/prometheus-cpp
git submodule add https://github.com/alibaba/PhotonLibOS.git archive/third_party/PhotonLibOS
# 2. 添加 kt-kernel/ 下的第三方库
git submodule add https://github.com/ggerganov/llama.cpp.git kt-kernel/third_party/llama.cpp
git submodule add https://github.com/pybind/pybind11.git kt-kernel/third_party/pybind11
# 切换分支
cd archive/third_party/custom_flashinfer
git checkout fix-precision-mla-merge-main
cd ../../..
# 初始化并更新所有子模块（包括嵌套的子模块）
git submodule update --init --recursive
```
- 对于arm平台，注释掉
```bash
./third_party/llamafile/iqk_mul_mat_arm82.cpp
```
中的
```bash
#define iqk_mul_mat iqk_mul_mat_arm82
#define iqk_mul_mat_moe iqk_mul_mat_moe_arm82
```
- 执行`source /usr/local/Ascend/ascend-toolkit/set_env.sh`（以实际CANN-TOOLKIT安装路径为准）。
- 执行`apt install cmake libhwloc-dev pkg-config`安装依赖。
- 修改项目目录下 /ktransformers/config/config.yaml 中attn部分的page_size: 128 chunk_size: 16384
- 执行`USE_BALANCE_SERVE=1 USE_NUMA=1 bash ./install.sh`，等待安装完成。
首次编译，发现手动添加的模块/ktransformers/archive/third_party/llama.cpp结构不符，
报错：
```bash
/usr/bin/c++ -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -I/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party -O3 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -std=gnu++17 -MD -MT CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_zen4.cpp.o -MF CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_zen4.cpp.o.d -o CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_zen4.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_zen4.cpp
/usr/bin/c++ -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -I/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party -O3 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -std=gnu++17 -MD -MT CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp.o -MF CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp.o.d -o CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp
In file included from /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_mixmul.inc:23,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_mixmul_arm82.cpp:8:
/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu.h:50:10: fatal error: llama.cpp/ggml-impl.h: No such file or directory
50 | #include "llama.cpp/ggml-impl.h"
| ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm.inc:23,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm.inc:50,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm80.cpp:8:
/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu.h:50:10: fatal error: llama.cpp/ggml-impl.h: No such file or directory
50 | #include "llama.cpp/ggml-impl.h"
| ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/llamafile.dir/build.make:244: CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_mixmul_arm82.cpp.o] Error 1
gmake[2]: *** Waiting for unfinished jobs....
gmake[2]: *** [CMakeFiles/llamafile.dir/build.make:342: CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm80.cpp.o] Error 1
In file included from /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_mixmul.inc:23,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_mixmul_arm80.cpp:8:
/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu.h:50:10: fatal error: llama.cpp/ggml-impl.h: No such file or directory
50 | #include "llama.cpp/ggml-impl.h"
| ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm.inc:23,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm.inc:50,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp:9:
/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu.h:50:10: fatal error: llama.cpp/ggml-impl.h: No such file or directory
50 | #include "llama.cpp/ggml-impl.h"
| ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
In file included from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat.inc:34,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm82.cpp:9:
/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:27:10: fatal error: llama.cpp/ggml-impl.h: No such file or directory
27 | #include "llama.cpp/ggml-impl.h"
| ^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
gmake[2]: *** [CMakeFiles/llamafile.dir/build.make:230: CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_mixmul_arm80.cpp.o] Error 1
gmake[2]: *** [CMakeFiles/llamafile.dir/build.make:356: CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp.o] Error 1
gmake[2]: *** [CMakeFiles/llamafile.dir/build.make:118: CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm82.cpp.o] Error 1
gmake[2]: Leaving directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
gmake[1]: *** [CMakeFiles/Makefile2:216: CMakeFiles/llamafile.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
[ 14%] Linking CXX shared library ../../../../bin/libggml-base.so
cd /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build/third_party/llama.cpp/ggml/src && /usr/bin/cmake -E cmake_link_script CMakeFiles/ggml-base.dir/link.txt --verbose=1
/usr/bin/c++ -fPIC -O3 -ffast-math -fopenmp -O2 -DNDEBUG -shared -Wl,-soname,libggml-base.so.0 -o ../../../../bin/libggml-base.so.0.9.4 CMakeFiles/ggml-base.dir/ggml.c.o CMakeFiles/ggml-base.dir/ggml.cpp.o CMakeFiles/ggml-base.dir/ggml-alloc.c.o CMakeFiles/ggml-base.dir/ggml-backend.cpp.o CMakeFiles/ggml-base.dir/ggml-opt.cpp.o CMakeFiles/ggml-base.dir/ggml-threading.cpp.o CMakeFiles/ggml-base.dir/ggml-quants.c.o CMakeFiles/ggml-base.dir/gguf.cpp.o -lm
cd /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build/third_party/llama.cpp/ggml/src && /usr/bin/cmake -E cmake_symlink_library ../../../../bin/libggml-base.so.0.9.4 ../../../../bin/libggml-base.so.0 ../../../../bin/libggml-base.so
gmake[2]: Leaving directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
[ 14%] Built target ggml-base
gmake[1]: Leaving directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
gmake: *** [Makefile:136: all] Error 2
CMake args: ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/user3/workspace/ktransformers/archive/build/lib.linux-aarch64-cpython-311/', '-DPYTHON_EXECUTABLE=/home/user3/miniconda3/envs/py311/bin/python3.11', '-DCMAKE_BUILD_TYPE=Release', '-DKTRANSFORMERS_USE_NPU=ON', '-DKTRANSFORMERS_USE_CUDA=OFF', '-D_GLIBCXX_USE_CXX11_ABI=0']
CMake args: ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/user3/workspace/ktransformers/archive/build/lib.linux-aarch64-cpython-311/', '-DPYTHON_EXECUTABLE=/home/user3/miniconda3/envs/py311/bin/python3.11', '-DCMAKE_BUILD_TYPE=Release', '-DKTRANSFORMERS_USE_NPU=ON', '-DKTRANSFORMERS_USE_CUDA=OFF', '-D_GLIBCXX_USE_CXX11_ABI=0', '-DLLAMA_NATIVE=ON', '-DEXAMPLE_VERSION_INFO=0.4.1+npu2.5.1.post3torch25aarch64']
build_temp: /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build
```
将仓库/kransformers/third_party下的llama.cpp模块覆盖掉/ktransformers/archive/third_party的llama.cpp模块，就能找到llama.cpp/ggml-impl.h头文件，再次编译，头文件可以找到，但是出现了新的问题：
```bash
[ 36%] Building CXX object CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_unsupported.cpp.o
/usr/bin/c++ -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -I/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party -O1 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -std=gnu++17 -MD -MT CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_mixmul_arm82.cpp.o -MF CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_mixmul_arm82.cpp.o.d -o CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_mixmul_arm82.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_mixmul_arm82.cpp
cd /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build/third_party/llama.cpp && /usr/bin/c++ -DGGML_SCHED_MAX_COPIES=4 -DGGML_USE_LLAMAFILE -DGGML_USE_OPENMP -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -D_GNU_SOURCE -D_XOPEN_SOURCE=600 -I/home/user3/workspace/ktransformers/archive/third_party/llama.cpp/. -O1 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -Wmissing-declarations -Wmissing-noreturn -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-array-bounds -Wno-format-truncation -Wextra-semi -fopenmp -std=gnu++11 -MD -MT third_party/llama.cpp/CMakeFiles/ggml.dir/sgemm.cpp.o -MF CMakeFiles/ggml.dir/sgemm.cpp.o.d -o CMakeFiles/ggml.dir/sgemm.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llama.cpp/sgemm.cpp
/usr/bin/c++ -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -I/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party -O1 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -std=gnu++17 -MD -MT CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx.cpp.o -MF CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx.cpp.o.d -o CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx.cpp
/usr/bin/c++ -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -I/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party -O1 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -std=gnu++17 -MD -MT CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx2.cpp.o -MF CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx2.cpp.o.d -o CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx2.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx2.cpp
/usr/bin/c++ -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -I/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party -O1 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -std=gnu++17 -MD -MT CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx512f.cpp.o -MF CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx512f.cpp.o.d -o CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx512f.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avx512f.cpp
/usr/bin/c++ -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -I/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party -O1 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -std=gnu++17 -MD -MT CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avxvnni.cpp.o -MF CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avxvnni.cpp.o.d -o CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avxvnni.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_avxvnni.cpp
/usr/bin/c++ -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -I/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party -O1 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -std=gnu++17 -MD -MT CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_fma.cpp.o -MF CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_fma.cpp.o.d -o CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_fma.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_fma.cpp
/usr/bin/c++ -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -I/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party -O1 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -std=gnu++17 -MD -MT CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm80.cpp.o -MF CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm80.cpp.o.d -o CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm80.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm80.cpp
/usr/bin/c++ -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -I/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party -O1 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -std=gnu++17 -MD -MT CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_zen4.cpp.o -MF CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_zen4.cpp.o.d -o CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_zen4.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_amd_zen4.cpp
/usr/bin/c++ -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -I/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party -O1 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -std=gnu++17 -MD -MT CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp.o -MF CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp.o.d -o CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_sgemm_arm82.cpp
/usr/bin/c++ -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -I/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party -O1 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -std=gnu++17 -MD -MT CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_unsupported.cpp.o -MF CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_unsupported.cpp.o.d -o CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_unsupported.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llamafile/tinyblas_cpu_unsupported.cpp
gmake[2]: Leaving directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
[ 37%] Built target build_info
In file included from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat.inc:34,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm82.cpp:9:
/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4135:24: warning: ‘always_inline’ function might not be inlinable [-Wattributes]
4135 | IQK_ALWAYS_INLINE void compute_8_blocks(const int8x16_t * qx, const Q8& q8,
| ^~~~~~~~~~~~~~~~
/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4135:24: warning: ‘always_inline’ function might not be inlinable [-Wattributes]
/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4135:24: warning: ‘always_inline’ function might not be inlinable [-Wattributes]
/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4135:24: warning: ‘always_inline’ function might not be inlinable [-Wattributes]
/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4135:24: warning: ‘always_inline’ function might not be inlinable [-Wattributes]
/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4135:24: warning: ‘always_inline’ function might not be inlinable [-Wattributes]
/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4135:24: warning: ‘always_inline’ function might not be inlinable [-Wattributes]
/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4135:24: warning: ‘always_inline’ function might not be inlinable [-Wattributes]
/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4112:24: warning: ‘always_inline’ function might not be inlinable [-Wattributes]
....ommit...
4953 | IQK_ALWAYS_INLINE int32x4_t prepare_scales_8(const uint32x4_t& v1) {
| ^~~~~~~~~~~~~~~~
/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4938:31: warning: ‘always_inline’ function might not be inlinable [-Wattributes]
4938 | IQK_ALWAYS_INLINE int32x4x2_t prepare_scales_8(const uint32x4_t& v1, const uint32x4_t& v2) {
| ^~~~~~~~~~~~~~~~
/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4664:24: warning: ‘always_inline’ function might not be inlinable [-Wattributes]
4664 | IQK_ALWAYS_INLINE void fusion_mul_mat_qX_K_q8_K_T_y1_d4k(
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4476:24: warning: ‘always_inline’ function might not be inlinable [-Wattributes]
4476 | IQK_ALWAYS_INLINE void fusion_mul_mat_qX_K_q8_K_T_y1_d6k(
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party/llama.cpp/ggml-impl.h:160,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:27,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat.inc:34,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm82.cpp:9:
In function ‘int32x4_t vdotq_laneq_s32(int32x4_t, int8x16_t, int8x16_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerIQ4XS>; int BN = 1]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:3986:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:33614:44: error: argument 4 must be a constant immediate
33614 | return __builtin_aarch64_sdot_laneqv16qi (__r, __a, __b, __index);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:33614:44: error: argument 4 must be a constant immediate
In file included from /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party/llama.cpp/ggml-impl.h:160,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:27,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat.inc:34,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm82.cpp:9:
In function ‘int32x4_t vmlal_laneq_s16(int32x4_t, int16x4_t, int16x8_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerIQ4XS>; int BN = 1]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4037:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:7749:49: error: argument 4 must be a constant immediate
7749 | return __builtin_aarch64_vec_smlal_laneq_v4hi (__acc, __a, __b, __c);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
In function ‘int32x4_t vmlal_laneq_s16(int32x4_t, int16x4_t, int16x8_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerIQ4XS>; int BN = 1]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4039:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:7749:49: error: argument 4 must be a constant immediate
7749 | return __builtin_aarch64_vec_smlal_laneq_v4hi (__acc, __a, __b, __c);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
In file included from /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party/llama.cpp/ggml-impl.h:160,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:27,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat.inc:34,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm82.cpp:9:
In function ‘int32x4_t vdotq_laneq_s32(int32x4_t, int8x16_t, int8x16_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerIQ4XS>; int BN = 2]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:3986:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:33614:44: error: argument 4 must be a constant immediate
.... omit....
33614 | return __builtin_aarch64_sdot_laneqv16qi (__r, __a, __b, __index);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:33614:44: error: argument 4 must be a constant immediate
In file included from /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party/llama.cpp/ggml-impl.h:160,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:27,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat.inc:34,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm82.cpp:9:
In function ‘int32x4_t vmlal_laneq_s16(int32x4_t, int16x4_t, int16x8_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerQ2K>; int BN = 3]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4037:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:7749:49: error: argument 4 must be a constant immediate
7749 | return __builtin_aarch64_vec_smlal_laneq_v4hi (__acc, __a, __b, __c);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
In function ‘int32x4_t vmlal_laneq_s16(int32x4_t, int16x4_t, int16x8_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerQ2K>; int BN = 3]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4039:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:7749:49: error: argument 4 must be a constant immediate
7749 | return __builtin_aarch64_vec_smlal_laneq_v4hi (__acc, __a, __b, __c);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
In file included from /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party/llama.cpp/ggml-impl.h:160,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:27,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat.inc:34,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm82.cpp:9:
In function ‘int32x4_t vdotq_laneq_s32(int32x4_t, int8x16_t, int8x16_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerQ6K>; int BN = 3]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:3986:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:33614:44: error: argument 4 must be a constant immediate
33614 | return __builtin_aarch64_sdot_laneqv16qi (__r, __a, __b, __index);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:33614:44: error: argument 4 must be a constant immediate
gmake[2]: Leaving directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
[ 37%] Built target ggml
/usr/bin/gmake -f third_party/llama.cpp/CMakeFiles/llama.dir/build.make third_party/llama.cpp/CMakeFiles/llama.dir/depend
/usr/bin/gmake -f third_party/llama.cpp/CMakeFiles/ggml_static.dir/build.make third_party/llama.cpp/CMakeFiles/ggml_static.dir/depend
gmake[2]: Entering directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
cd /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext /home/user3/workspace/ktransformers/archive/third_party/llama.cpp /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build/third_party/llama.cpp /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build/third_party/llama.cpp/CMakeFiles/ggml_static.dir/DependInfo.cmake --color=
gmake[2]: Entering directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
cd /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build && /usr/bin/cmake -E cmake_depends "Unix Makefiles" /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext /home/user3/workspace/ktransformers/archive/third_party/llama.cpp /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build/third_party/llama.cpp /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build/third_party/llama.cpp/CMakeFiles/llama.dir/DependInfo.cmake --color=
In file included from /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/../../third_party/llama.cpp/ggml-impl.h:160,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:27,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat.inc:34,
from /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm82.cpp:9:
In function ‘int32x4_t vmlal_laneq_s16(int32x4_t, int16x4_t, int16x8_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerQ3K>; int BN = 4]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4037:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:7749:49: error: argument 4 must be a constant immediate
7749 | return __builtin_aarch64_vec_smlal_laneq_v4hi (__acc, __a, __b, __c);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
In function ‘int32x4_t vmlal_laneq_s16(int32x4_t, int16x4_t, int16x8_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerQ3K>; int BN = 4]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4039:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:7749:49: error: argument 4 must be a constant immediate
7749 | return __builtin_aarch64_vec_smlal_laneq_v4hi (__acc, __a, __b, __c);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
gmake[2]: Leaving directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
gmake[2]: Leaving directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
/usr/bin/gmake -f third_party/llama.cpp/CMakeFiles/llama.dir/build.make third_party/llama.cpp/CMakeFiles/llama.dir/build
/usr/bin/gmake -f third_party/llama.cpp/CMakeFiles/ggml_static.dir/build.make third_party/llama.cpp/CMakeFiles/ggml_static.dir/build
gmake[2]: Entering directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
gmake[2]: Entering directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
[ 42%] Building CXX object third_party/llama.cpp/CMakeFiles/llama.dir/llama.cpp.o
[ 42%] Building CXX object third_party/llama.cpp/CMakeFiles/llama.dir/unicode-data.cpp.o
[ 42%] Linking CXX static library libggml_static.a
[ 42%] Building CXX object third_party/llama.cpp/CMakeFiles/llama.dir/unicode.cpp.o
cd /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build/third_party/llama.cpp && /usr/bin/cmake -P CMakeFiles/ggml_static.dir/cmake_clean_target.cmake
cd /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build/third_party/llama.cpp && /usr/bin/c++ -DGGML_SCHED_MAX_COPIES=4 -DGGML_USE_LLAMAFILE -DGGML_USE_OPENMP -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -D_GNU_SOURCE -D_XOPEN_SOURCE=600 -I/home/user3/workspace/ktransformers/archive/third_party/llama.cpp/. -O1 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -Wmissing-declarations -Wmissing-noreturn -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-array-bounds -Wno-format-truncation -Wextra-semi -fopenmp -std=gnu++11 -MD -MT third_party/llama.cpp/CMakeFiles/llama.dir/llama.cpp.o -MF CMakeFiles/llama.dir/llama.cpp.o.d -o CMakeFiles/llama.dir/llama.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llama.cpp/llama.cpp
cd /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build/third_party/llama.cpp && /usr/bin/c++ -DGGML_SCHED_MAX_COPIES=4 -DGGML_USE_LLAMAFILE -DGGML_USE_OPENMP -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -D_GNU_SOURCE -D_XOPEN_SOURCE=600 -I/home/user3/workspace/ktransformers/archive/third_party/llama.cpp/. -O1 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -Wmissing-declarations -Wmissing-noreturn -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-array-bounds -Wno-format-truncation -Wextra-semi -fopenmp -std=gnu++11 -MD -MT third_party/llama.cpp/CMakeFiles/llama.dir/unicode.cpp.o -MF CMakeFiles/llama.dir/unicode.cpp.o.d -o CMakeFiles/llama.dir/unicode.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llama.cpp/unicode.cpp
cd /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build/third_party/llama.cpp && /usr/bin/c++ -DGGML_SCHED_MAX_COPIES=4 -DGGML_USE_LLAMAFILE -DGGML_USE_OPENMP -DKTRANSFORMERS_USE_NPU=1 -D_GLIBCXX_USE_CXX11_ABI=0 -D_GNU_SOURCE -D_XOPEN_SOURCE=600 -I/home/user3/workspace/ktransformers/archive/third_party/llama.cpp/. -O1 -ffast-math -fopenmp -O2 -DNDEBUG -fPIC -march=armv8.2-a+fp16+fp16fml+dotprod -lnuma -Wmissing-declarations -Wmissing-noreturn -Wall -Wextra -Wpedantic -Wcast-qual -Wno-unused-function -Wno-array-bounds -Wno-format-truncation -Wextra-semi -fopenmp -std=gnu++11 -MD -MT third_party/llama.cpp/CMakeFiles/llama.dir/unicode-data.cpp.o -MF CMakeFiles/llama.dir/unicode-data.cpp.o.d -o CMakeFiles/llama.dir/unicode-data.cpp.o -c /home/user3/workspace/ktransformers/archive/third_party/llama.cpp/unicode-data.cpp
cd /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build/third_party/llama.cpp && /usr/bin/cmake -E cmake_link_script CMakeFiles/ggml_static.dir/link.txt --verbose=1
/usr/bin/ar qc libggml_static.a CMakeFiles/ggml.dir/ggml.c.o CMakeFiles/ggml.dir/ggml-alloc.c.o CMakeFiles/ggml.dir/ggml-backend.c.o CMakeFiles/ggml.dir/ggml-quants.c.o CMakeFiles/ggml.dir/sgemm.cpp.o
/usr/bin/ranlib libggml_static.a
gmake[2]: Leaving directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
In function ‘int32x4_t vmlal_laneq_s16(int32x4_t, int16x4_t, int16x8_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerQ2K>; int BN = 4]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4037:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:7749:49: error: argument 4 must be a constant immediate
7749 | return __builtin_aarch64_vec_smlal_laneq_v4hi (__acc, __a, __b, __c);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
In function ‘int32x4_t vmlal_laneq_s16(int32x4_t, int16x4_t, int16x8_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerQ2K>; int BN = 4]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4039:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:7749:49: error: argument 4 must be a constant immediate
7749 | return __builtin_aarch64_vec_smlal_laneq_v4hi (__acc, __a, __b, __c);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
[ 42%] Built target ggml_static
In function ‘int32x4_t vmlal_laneq_s16(int32x4_t, int16x4_t, int16x8_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerIQ4XS>; int BN = 4]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4037:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:7749:49: error: argument 4 must be a constant immediate
7749 | return __builtin_aarch64_vec_smlal_laneq_v4hi (__acc, __a, __b, __c);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
In function ‘int32x4_t vmlal_laneq_s16(int32x4_t, int16x4_t, int16x8_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerIQ4XS>; int BN = 4]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4039:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:7749:49: error: argument 4 must be a constant immediate
7749 | return __builtin_aarch64_vec_smlal_laneq_v4hi (__acc, __a, __b, __c);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
In function ‘int32x4_t vmlal_laneq_s16(int32x4_t, int16x4_t, int16x8_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerQ5K>; int BN = 4]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4037:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:7749:49: error: argument 4 must be a constant immediate
7749 | return __builtin_aarch64_vec_smlal_laneq_v4hi (__acc, __a, __b, __c);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
In function ‘int32x4_t vmlal_laneq_s16(int32x4_t, int16x4_t, int16x8_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerQ5K>; int BN = 4]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4039:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:7749:49: error: argument 4 must be a constant immediate
7749 | return __builtin_aarch64_vec_smlal_laneq_v4hi (__acc, __a, __b, __c);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
In function ‘int32x4_t vmlal_laneq_s16(int32x4_t, int16x4_t, int16x8_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerQ4K>; int BN = 4]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4037:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:7749:49: error: argument 4 must be a constant immediate
7749 | return __builtin_aarch64_vec_smlal_laneq_v4hi (__acc, __a, __b, __c);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
In function ‘int32x4_t vmlal_laneq_s16(int32x4_t, int16x4_t, int16x8_t, int)’,
inlined from ‘void {anonymous}::matmul_v2_kernel(const Dequantizer*, const block_q8_K**, const {anonymous}::DataInfo&, int, int) [with Dequantizer = {anonymous}::BlockQxK<{anonymous}::DequantizerQ4K>; int BN = 4]’ at /home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm.inc:4039:64:
/usr/lib/gcc/aarch64-linux-gnu/10.3.1/include/arm_neon.h:7749:49: error: argument 4 must be a constant immediate
7749 | return __builtin_aarch64_vec_smlal_laneq_v4hi (__acc, __a, __b, __c);
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~
gmake[2]: *** [CMakeFiles/llamafile.dir/build.make:118: CMakeFiles/llamafile.dir/home/user3/workspace/ktransformers/archive/third_party/llamafile/iqk_mul_mat_arm82.cpp.o] Error 1
gmake[2]: Leaving directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
gmake[1]: *** [CMakeFiles/Makefile2:170: CMakeFiles/llamafile.dir/all] Error 2
gmake[1]: *** Waiting for unfinished jobs....
[ 44%] Linking CXX static library libllama.a
cd /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build/third_party/llama.cpp && /usr/bin/cmake -P CMakeFiles/llama.dir/cmake_clean_target.cmake
cd /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build/third_party/llama.cpp && /usr/bin/cmake -E cmake_link_script CMakeFiles/llama.dir/link.txt --verbose=1
/usr/bin/ar qc libllama.a CMakeFiles/llama.dir/llama.cpp.o CMakeFiles/llama.dir/unicode.cpp.o CMakeFiles/llama.dir/unicode-data.cpp.o CMakeFiles/ggml.dir/ggml.c.o CMakeFiles/ggml.dir/ggml-alloc.c.o CMakeFiles/ggml.dir/ggml-backend.c.o CMakeFiles/ggml.dir/ggml-quants.c.o CMakeFiles/ggml.dir/sgemm.cpp.o
/usr/bin/ranlib libllama.a
gmake[2]: Leaving directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
[ 44%] Built target llama
gmake[1]: Leaving directory '/home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build'
gmake: *** [Makefile:136: all] Error 2
CMake args: ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/user3/workspace/ktransformers/archive/build/lib.linux-aarch64-cpython-311/', '-DPYTHON_EXECUTABLE=/home/user3/miniconda3/envs/py311/bin/python3.11', '-DCMAKE_BUILD_TYPE=Release', '-DKTRANSFORMERS_USE_NPU=ON', '-DKTRANSFORMERS_USE_CUDA=OFF', '-D_GLIBCXX_USE_CXX11_ABI=0']
CMake args: ['-DCMAKE_LIBRARY_OUTPUT_DIRECTORY=/home/user3/workspace/ktransformers/archive/build/lib.linux-aarch64-cpython-311/', '-DPYTHON_EXECUTABLE=/home/user3/miniconda3/envs/py311/bin/python3.11', '-DCMAKE_BUILD_TYPE=Release', '-DKTRANSFORMERS_USE_NPU=ON', '-DKTRANSFORMERS_USE_CUDA=OFF', '-D_GLIBCXX_USE_CXX11_ABI=0', '-DLLAMA_NATIVE=ON', '-DEXAMPLE_VERSION_INFO=0.4.1+npu2.5.1.post3torch25aarch64']
build_temp: /home/user3/workspace/ktransformers/archive/csrc/ktransformers_ext/build
Traceback (most recent call last):
File "/home/user3/miniconda3/envs/py311/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 389, in
main()
File "/home/user3/miniconda3/envs/py311/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 373, in main
json_out["return_val"] = hook(**hook_input["kwargs"])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user3/miniconda3/envs/py311/lib/python3.11/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 280, in build_wheel
return _build_backend().build_wheel(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user3/miniconda3/envs/py311/lib/python3.11/site-packages/setuptools/build_meta.py", line 435, in build_wheel
return _build(['bdist_wheel', '--dist-info-dir', str(metadata_directory)])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user3/miniconda3/envs/py311/lib/python3.11/site-packages/setuptools/build_meta.py", line 423, in _build
return self._build_with_temp_dir(
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user3/miniconda3/envs/py311/lib/python3.11/site-packages/setuptools/build_meta.py", line 404, in _build_with_temp_dir
self.run_setup()
File "/home/user3/miniconda3/envs/py311/lib/python3.11/site-packages/setuptools/build_meta.py", line 317, in run_setup
exec(code, locals())
```
### Others
这是什么问题，我的ARM CPU经过验证支持 armv8.2-a + fp16 + fp16fml + dotprod，也正确开启了-DKTRANSFORMERS_USE_NPU=1，是什么原因呢，是否是GCC编译器版本太低的问题。我想了解这个部署Ascend NPU官方文档更详细的硬件配置和编译器配置。
Contributor guide
Assessment
This issue has not been assessed yet.