abetlen / abetlen/llama-cpp-python

Failed to run on Intel GPUs

Open
#1,268 2 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10.6k
Forks
1.4k
PR merge metrics
PR metrics pending

Description

# Prerequisites

Please answer the following questions for yourself before submitting an issue.

- [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [x] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).
- [ ] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
- [ ] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.

# Expected Behavior

I expect llama-cpp-python can normally run on Intel GPUs as llama.cpp do.

# Current Behavior

llama-cpp-python fail to run on Intel GPUs while llama.cpp sycl backend can run normally.

# Environment and Context

I test SYCL support on Intel Arc A770 GPU, with ubuntu 22.04 system, oneapi version is 2024.0 . I have verified llama.cpp sycl backend works normally on my machine.

# Steps to Reproduce

```
conda create -n llm python=3.9
conda activate llm
source /opt/intel/oneapi/setvars.sh
CMAKE_ARGS="-DLLAMA_SYCL=on -DCMAKE_C_COMPILER=icx -DCMAKE_CXX_COMPILER=icpx" pip install llama-cpp-python
python test.py
```
while test.py is
```python
from llama_cpp import Llama
llm = Llama(
model_path="~/llama.cpp/models/7B/ggml-model-q4_0-pure.gguf",
n_gpu_layers=33, # Uncomment to use GPU acceleration
seed=1337, # Uncomment to set a specific seed
# n_ctx=2048, # Uncomment to increase the context window
)
output = llm(
"Q: Name the planets in the solar system? A: ", # Prompt
max_tokens=32, # Generate up to 32 tokens, set to None to generate up to the end of the context window
stop=["Q:", "\n"], # Stop generating just before the model would generate a new question
echo=True # Echo the prompt back in the output
) # Generate a completion, can also call create_completion
print(output)
```

# Failure Logs

```bash
ggml_init_sycl: GGML_SYCL_DEBUG: 0
ggml_init_sycl: GGML_SYCL_F16: no
found 2 SYCL devices:
|ID| Name |compute capability|Max compute units|Max work group|Max sub group|Global mem size|
|--|---------------------------------------------|------------------|-----------------|--------------|-------------|---------------|
| 0| 13th Gen Intel(R) Core(TM) i9-13900K| 3.0| 32| 8192| 64| 67181625344|
| 1| Intel(R) FPGA Emulation Device| 1.2| 32| 67108864| 64| 67181625344|
DeviceList is empty. -30 (PI_ERROR_INVALID_VALUE)Exception caught at file:/tmp/pip-install-31terybs/llama-cpp-python_2e42ff812a094f19b998956fddc30615/vendor/llama.cpp/ggml-sycl.cpp, line:13341
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.