abetlen / abetlen/llama-cpp-python

cuBLAS error 15

Open
#941 2 comments 1 reaction 0 assignees View on GitHub
bug
Dominant language
Python
Stars
10.6k
Forks
1.4k
PR merge metrics
PR metrics pending

Description

# Prerequisites

Please answer the following questions for yourself before submitting an issue.

- [x] I am running the latest code. Development is very rapid so there are no tagged versions as of now.
- [x] I carefully followed the [README.md](https://github.com/abetlen/llama-cpp-python/blob/main/README.md).
- [x] I [searched using keywords relevant to my issue](https://docs.github.com/en/issues/tracking-your-work-with-issues/filtering-and-searching-issues-and-pull-requests) to make sure that I am creating a new issue that is not already open (or closed).
- [x] I reviewed the [Discussions](https://github.com/abetlen/llama-cpp-python/discussions), and have a new bug or useful enhancement to share.

# Expected Behavior

I am trying to run [privateGPT](https://github.com/imartinez/privateGPT) with CUDA.
Without CUDA it works. With CUDA enabled, 'llama-cpp-python' crashes.

# Current Behavior
When I start privateGPT, llama-cpp-python-main gives this error : ```cuBLAS error 15 at C:\Users\Harry\Documents\llama-cpp-python-main\vendor\llama.cpp\ggml-cuda.cu:7586: the requested functionality is not supported```

# Environment and Context

I am on Windows 11, with NVIDIA CUDA Toolkit 12.2 installed, NVIDIA GeForce GTX 960 4GB and 32GB RAM. CPU is Intel i7 2600K (it should support AVX).

```
$ python3 --version
$ Python 3.11.6

$ make --version
$ GNU Make 4.4.1

$ g++ --version
$ g++ (x86_64-posix-seh-rev0, Built by MinGW-Builds project) 13.2.0
```

# Failure Information (for bugs)
```cuBLAS error 15 at C:\Users\Harry\Documents\llama-cpp-python-main\vendor\llama.cpp\ggml-cuda.cu:7586: the requested functionality is not supported```

# Steps to Reproduce

```
git clone https://github.com/imartinez/privateGPT
cd privateGPT

conda create -n privateGPT python=3.11
conda activate privateGPT

poetry install --with ui,local

poetry run python scripts/setup

cd ..

git clone --recurse-submodules https://github.com/abetlen/llama-cpp-python.git llama-cpp-python-main
cd llama-cpp-python-main

set FORCE_CMAKE=1
set CMAKE_ARGS=-DLLAMA_CUBLAS=on -DLLAMA_AVX=on -DLLAMA_AVX2=off -DLLAMA_FMA=off
python -m pip install .[all]

cd ..
cd privateGPT
make run
```

Let me know if I can give you more info, thank you.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.