abetlen / abetlen/llama-cpp-python

Metal installation documentation

Offen
#1,968 0 Kommentare 3 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
10.6k
Forks
1.4k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

I tried setting up `llama-cpp-python` in the current version `0.3.7` on my MacBook M4 Pro.
In the first step I only installed via `pip install llama-cpp-python --no-cache-dir` without specifiying the environment variable for Metal backend support.
I set the `n_gpu_layers` to `-1` to fully use the GPU.

The interesting thing is the GPU was used even without having to install the Metal backend support as stated in the current documentation. I double checked this with a fresh start and explicitely setting the `CMAKE_ARGS` env variable and did not see and difference in terms of performance or GPU usage.

This is pretty handy, because when not using `pip` for dependency management (e.g. `poetry`) passing the environment variable did not work on my side.

Maybe the documentation should be updated to state that the env arguments are no longer required? This would also reflect the documentation in https://github.com/ggml-org/llama.cpp/blob/master/docs/build.md#metal-build where it states that: "On MacOS, Metal is enabled by default"

This would also mean, that custom pre-built wheels are no longer required as well.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.