abetlen / abetlen/llama-cpp-python

ERROR installing v0.3.16 with CUDA enabled on docker

Abierto
#2,061 3 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Python
Estrellas
10.6k
Forks
1.4k
Métricas de merge de PR
Métricas de PR pendientes

Descripción

```
# takes build time + 5-8 minutes to complete
FROM nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive
ENV HF_TOKEN=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
ENV TZ=Asia/Hong_Kong

# install linux packages
RUN apt-get update && \
apt-get update && apt-get install -y sudo && \
apt-get update && apt-get install -y nano

# install python
RUN apt-get install -y python3-pip python3-dev
RUN apt-get install cmake -y
RUN apt-get install git -y

# install CUDA env
RUN apt-get install cuda-toolkit-12-4 -y
# RUN pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124
RUN pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
RUN pip install torch-cluster -f https://data.pyg.org/whl/torch-2.5.1+cu124.html

# necessary to install llama-cpp-python
RUN apt-get update && \
apt-get install -y \
ninja-build

# install llama-cpp-python with CUDA enabled
ENV GGML_CUDA=1
ENV FORCE_CMAKE=1
ENV CMAKE_ARGS=-DGGML_CUDA=on

ENV CMAKE_ARGS=-DCMAKE_GENERATOR_TOOLSET="cuda=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4"
# dpkg -S libcuda.so.1
ENV LD_LIBRARY_PATH=/usr/local/cuda-12.4/compat/libcuda.so

RUN CMAKE_ARGS="-DGGML_CUDA=on" pip install --user llama-cpp-python==0.3.16 \
--extra-index-url https://github.com/abetlen/llama-cpp-python/releases/download/v0.3.16-cu124/llama_cpp_python-0.3.16-cp310-cp310-linux_x86_64.whl \
--verbose
```

Hi, I am trying to install llama-cpp-python with GPU enabled. It worked for v0.2.77, but I need a more recent version. The issue is that by using v0.3.16 I had to use CMAKE_ARGS="-DGGML_CUDA=on" instead of using CMAKE_ARGS="-DLLAMA_CUBLAS=on" (that's the only change I was forced to do).
The build gives me the following error: I searched, and one solution (https://github.com/abetlen/llama-cpp-python/issues/1617) was to add the LD_LIBRARY_PATH (tried with libcuda.so and libcuda.so.1), but still the same issue.

Also, is there an easier way (that perhaps I missed) to install v0.3.16?
Thank you
```
ERROR:
/usr/bin/ld: warning: libcuda.so.1, needed by bin/libggml-cuda.so, not found (try using -rpath or -rpath-link)
```

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.