mudler / mudler/LocalAI

docker-compose.yaml GPU example: capabilities [gpu, utility] omits compute, so libcuda.so.1 is not injected

Open Beginner friendly
#11,973 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
49.2k
Forks
4.5k
Avg merge
1d 3m
Merged PRs (30d)
239

Description

LocalAI version:
quay.io/go-skynet/local-ai:master-gpu-nvidia-cuda-13 (sha256:02e823f5f25ddef3f4edb94a8960e506daa4f71595d7c75b041d52b6eaaf425b, LocalAI bcf196d), docker-compose.yaml at the same commit.

Environment, CPU architecture, OS, and Version:
Docker Desktop 29.7.2 (Compose 5.5.1) on Windows 11, WSL2 kernel 6.18.33.2-microsoft-standard-WSL2, x86_64. NVIDIA RTX 5070 Ti, driver 616.92.

Describe the bug
The commented NVIDIA example in docker-compose.yaml recommends:

    # environment:
    #   NVIDIA_DRIVER_CAPABILITIES: "compute,utility"
    # init: true
    # deploy:
    #   resources:
    #     reservations:
    #       devices:
    #         - driver: nvidia
    #           count: 1
    #           capabilities: [gpu, utility]

With capabilities: [gpu, utility], the container gets only the utility driver libraries. docker inspect shows "Capabilities":[["gpu","utility"]], and /usr/lib/x86_64-linux-gnu contains libnvidia-ml.so.1 and libdxcore.so but no libcuda.so.1. nvidia-smi works inside the container, so the setup looks fine, but every CUDA backend fails to start with ImportError: libcuda.so.1: cannot open shared object file. In my setup, also setting NVIDIA_DRIVER_CAPABILITIES=compute,utility in environment: (as the example suggests) did not change this.

Changing it to capabilities: [gpu, compute, utility] fixes it. libcuda.so.1 then shows up in /usr/lib/x86_64-linux-gnu and in ldconfig -p, and CUDA backends load.

To Reproduce

  1. Enable the legacy driver: nvidia example from docker-compose.yaml as written, with a CUDA image such as master-gpu-nvidia-cuda-13.
  2. docker compose up -d
  3. docker exec <container> sh -c 'ldconfig -p | grep libcuda.so.1' returns nothing. Loading a model on a Python CUDA backend fails with the libcuda.so.1 error above.

Expected behavior
Following the compose example gives a container where CUDA backends work.

Logs

ERROR Failed to load model ... error=failed to load model with internal loader: grpc service not ready: backend process exited with code 1: ImportError: libcuda.so.1: cannot open shared object file: No such file or directory

Additional context
Suggested fix: use capabilities: [gpu, compute, utility] in the examples, and mention that compute is what brings in libcuda. I only tested the legacy driver: nvidia variant, on Docker Desktop/WSL2. I did not test the CDI (nvidia.com/gpu) variant, which may treat capabilities differently. It's worth checking the docs pages that show the same snippet as well.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the commented NVIDIA example in docker-compose.yaml and search the documentation pages for the same capabilities snippet. Reproduce the legacy driver variant if possible, then update the examples so CUDA backends receive libcuda.so.1; verify the compose configuration includes compute alongside gpu and utility.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, docker-compose
Domain
devops, documentation
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.