abetlen / abetlen/llama-cpp-python

Can't install GPU version for windows for many times.

Đang mở
#1,393 10 bình luận 2 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
10.6k
Fork
1.4k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

## Issues
I am trying to install the lastest version of llama-cpp-python in my windows 11 with RTX-3090ti(24G). I have successfully installed llama-cpp-python=0.1.87 (can't exactly remember) months ago while using:
``` conda prompt
set FORCE_CMAKE=1
set CMAKE_ARGS=-DLLAMA_CUBLAS=on
pip install --upgrade --force-reinstall llama-cpp-python --no-cache-dir
```
But when I want to access the latest version recently by using:
``` conda prompt
set CMAKE_ARGS="-DLLAMA_CUDA=on"
pip install --upgrade --force-reinstall llama-cpp-python --no-cache-dir
```
After loading the model, it is still using CPU with `BLAS=0` (or is another params = 1 instead of `BLAS` in new version?).
```
llama_new_context_with_model: n_ctx = 2048
llama_new_context_with_model: n_batch = 1024
llama_new_context_with_model: n_ubatch = 512
llama_new_context_with_model: freq_base = 500000.0
llama_new_context_with_model: freq_scale = 1
llama_kv_cache_init: CPU KV buffer size = 256.00 MiB
llama_new_context_with_model: KV self size = 256.00 MiB, K (f16): 128.00 MiB, V (f16): 128.00 MiB
llama_new_context_with_model: CPU output buffer size = 0.49 MiB
llama_new_context_with_model: CPU compute buffer size = 258.50 MiB
llama_new_context_with_model: graph nodes = 1030
llama_new_context_with_model: graph splits = 1
AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 0 | SSE3 = 1 | SSSE3 = 0 | VSX = 0 | MATMUL_INT8 = 0 | LAMMAFILE = 1 |
Model metadata: {'general.name': 'Meta-Llama-3-8B-Instruct-imatrix', 'general.architecture': 'llama', 'llama.block_count': '32', 'llama.context_length': '8192', 'tokenizer.ggml.eos_token_id': '128001', 'general.file_type': '18', 'llama.attention.head_count_kv': '8', 'llama.embedding_length': '4096', 'llama.feed_forward_length': '14336', 'llama.attention.head_count': '32', 'llama.rope.freq_base': '500000.000000', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.vocab_size': '128256', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.model': 'gpt2', 'general.quantization_version': '2', 'tokenizer.ggml.bos_token_id': '128000', 'tokenizer.chat_template': "{% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>\n\n'+ message['content'] | trim + '<|eot_id|>' %}{% if loop.index0 == 0 %}{% set content = bos_token + content %}{% endif %}{{ content }}{% endfor %}{{ '<|start_header_id|>assistant<|end_header_id|>\n\n' }}"}
Using gguf chat template: {% set loop_messages = messages %}{% for message in loop_messages %}{% set content = '<|start_header_id|>' + message['role'] + '<|end_header_id|>
```
I have been trying the pre-build wheel for CUDA 12.1 (`pip install llama-cpp-python --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cu121`) and it still doesn't work. I add `--verbose` to see the output:

```
loading initial cache file C:\Users\Administrator\AppData\Local\Temp\tmpbbvy3nqu\build\CMakeInit.txt
-- Building for: Visual Studio 17 2022
-- Selecting Windows SDK version 10.0.22621.0 to target Windows 10.0.22631.
-- The C compiler identification is MSVC 19.39.33523.0
-- The CXX compiler identification is MSVC 19.39.33523.0
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: F:/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: F:/Microsoft Visual Studio/2022/Community/VC/Tools/MSVC/14.39.33519/bin/Hostx64/x64/cl.exe - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Found Git: F:/Git/cmd/git.exe (found version "2.44.0.windows.1")
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
-- Found Threads: TRUE
-- Warning: ccache not found - consider installing it for faster compilation or disable this warning with LLAMA_CCACHE=OFF
-- CMAKE_SYSTEM_PROCESSOR: AMD64
-- CMAKE_GENERATOR_PLATFORM: x64
-- x86 detected
-- Performing Test HAS_AVX_1
-- Performing Test HAS_AVX_1 - Success
-- Performing Test HAS_AVX2_1
-- Performing Test HAS_AVX2_1 - Success
-- Performing Test HAS_FMA_1
-- Performing Test HAS_FMA_1 - Success
-- Performing Test HAS_AVX512_1
-- Performing Test HAS_AVX512_1 - Failed
-- Performing Test HAS_AVX512_2
-- Performing Test HAS_AVX512_2 - Failed

```
## Environment
python=3.12
C++ compiler: viusal studio 2022 (with necessary C++ modules)
`cmake --version` = 3.29.2
`nvcc -V` = CUDA 12.1 (while `nvidia-smi` cuda version is 12.3, i think it is not related to this issues)

I have been download and install VS2022, CUDA toolkit, cmake and anaconda, I am wondering if some steps are missing. Considering my previous experience there is no need to git clone this code and cd into it to build (Though I did that on my mac to build a pth file to bin file months ago).

My system variables are listed below:
- F:\Anaconda\Scripts
- F:\CMake\bin
- C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1
- C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.1\bin

## Questions
1. Are there some steps i have been missing for build the llama-cpp for GPU version?
2. How to know if it is built for GPU version when i run `pip install llama-cpp-python` instead of loading model to check `BLAS=1.`
3. Do I need to git clone this code and cd into some dir, create some file or dir to run the `pip install llama-cpp-py`?

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.