abetlen / abetlen/llama-cpp-python

RPC is broken due to change of interface in llama.cpp main repository (rpc : early register backend devices #11262)

Open
#1,914 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10.6k
Forks
1.4k
PR merge metrics
PR metrics pending

Description

RPC does not work in llama-cpp-python since this commit in [llama.cpp](https://github.com/ggerganov/llama.cpp)
```
Revision: 667d72846c06b2cf4f7c8a4265e210991a49706b
Author: Radoslav Gerganov
Date: 17/01/2025 09:57:09
Message:
rpc : early register backend devices (#11262)
Early register RPC devices and do not propagate RPC specifics in the
llama model structures.

This commit change the way rpc_servers are passed to llama.cpp, (in llama.cpp\common\arg.cpp) it is not a parameter anymore (params.rpc_servers = value;) but need to call add_rpc_devices(value);
```

To reproduce:
```
from llama_cpp import Llama
llm = Llama(
model_path=MODEL,
rpc_servers=RPC_NODES,
n_ctx = 4096,
flash_attn = False,
n_gpu_layers = 99,
verbose=True
)
```
--> should init with RPC offloading

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.