abetlen / abetlen/llama-cpp-python

RPC is broken due to change of interface in llama.cpp main repository (rpc : early register backend devices #11262)

Aperta
#1,914 5 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
10.6k
Fork
1.4k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

RPC does not work in llama-cpp-python since this commit in [llama.cpp](https://github.com/ggerganov/llama.cpp)
```
Revision: 667d72846c06b2cf4f7c8a4265e210991a49706b
Author: Radoslav Gerganov
Date: 17/01/2025 09:57:09
Message:
rpc : early register backend devices (#11262)
Early register RPC devices and do not propagate RPC specifics in the
llama model structures.

This commit change the way rpc_servers are passed to llama.cpp, (in llama.cpp\common\arg.cpp) it is not a parameter anymore (params.rpc_servers = value;) but need to call add_rpc_devices(value);
```

To reproduce:
```
from llama_cpp import Llama
llm = Llama(
model_path=MODEL,
rpc_servers=RPC_NODES,
n_ctx = 4096,
flash_attn = False,
n_gpu_layers = 99,
verbose=True
)
```
--> should init with RPC offloading

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.