abetlen / abetlen/llama-cpp-python

ERROR installing v0.3.16 with CUDA enabled on docker

Đang mở
#2,061 3 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
10.6k
Fork
1.4k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

```
# takes build time + 5-8 minutes to complete
FROM nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04

ENV DEBIAN_FRONTEND=noninteractive
ENV HF_TOKEN=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
ENV TZ=Asia/Hong_Kong

# install linux packages
RUN apt-get update && \
apt-get update && apt-get install -y sudo && \
apt-get update && apt-get install -y nano

# install python
RUN apt-get install -y python3-pip python3-dev
RUN apt-get install cmake -y
RUN apt-get install git -y

# install CUDA env
RUN apt-get install cuda-toolkit-12-4 -y
# RUN pip install torch==2.5.1 torchvision==0.20.1 torchaudio==2.5.1 --index-url https://download.pytorch.org/whl/cu124
RUN pip install torch==2.6.0 torchvision==0.21.0 torchaudio==2.6.0 --index-url https://download.pytorch.org/whl/cu124
RUN pip install torch-cluster -f https://data.pyg.org/whl/torch-2.5.1+cu124.html

# necessary to install llama-cpp-python
RUN apt-get update && \
apt-get install -y \
ninja-build

# install llama-cpp-python with CUDA enabled
ENV GGML_CUDA=1
ENV FORCE_CMAKE=1
ENV CMAKE_ARGS=-DGGML_CUDA=on

ENV CMAKE_ARGS=-DCMAKE_GENERATOR_TOOLSET="cuda=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.4"
# dpkg -S libcuda.so.1
ENV LD_LIBRARY_PATH=/usr/local/cuda-12.4/compat/libcuda.so

RUN CMAKE_ARGS="-DGGML_CUDA=on" pip install --user llama-cpp-python==0.3.16 \
--extra-index-url https://github.com/abetlen/llama-cpp-python/releases/download/v0.3.16-cu124/llama_cpp_python-0.3.16-cp310-cp310-linux_x86_64.whl \
--verbose
```

Hi, I am trying to install llama-cpp-python with GPU enabled. It worked for v0.2.77, but I need a more recent version. The issue is that by using v0.3.16 I had to use CMAKE_ARGS="-DGGML_CUDA=on" instead of using CMAKE_ARGS="-DLLAMA_CUBLAS=on" (that's the only change I was forced to do).
The build gives me the following error: I searched, and one solution (https://github.com/abetlen/llama-cpp-python/issues/1617) was to add the LD_LIBRARY_PATH (tried with libcuda.so and libcuda.so.1), but still the same issue.

Also, is there an easier way (that perhaps I missed) to install v0.3.16?
Thank you
```
ERROR:
/usr/bin/ld: warning: libcuda.so.1, needed by bin/libggml-cuda.so, not found (try using -rpath or -rpath-link)
```

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.