abetlen / abetlen/llama-cpp-python
RPC is broken due to change of interface in llama.cpp main repository (rpc : early register backend devices #11262)
- 主要语言
- Python
- 星标
- 10.6k
- 派生
- 1.4k
- PR 合并指标
- PR 指标待抓取
描述
RPC does not work in llama-cpp-python since this commit in [llama.cpp](https://github.com/ggerganov/llama.cpp)
```
Revision: 667d72846c06b2cf4f7c8a4265e210991a49706b
Author: Radoslav Gerganov
Date: 17/01/2025 09:57:09
Message:
rpc : early register backend devices (#11262)
Early register RPC devices and do not propagate RPC specifics in the
llama model structures.
This commit change the way rpc_servers are passed to llama.cpp, (in llama.cpp\common\arg.cpp) it is not a parameter anymore (params.rpc_servers = value;) but need to call add_rpc_devices(value);
```
To reproduce:
```
from llama_cpp import Llama
llm = Llama(
model_path=MODEL,
rpc_servers=RPC_NODES,
n_ctx = 4096,
flash_attn = False,
n_gpu_layers = 99,
verbose=True
)
```
--> should init with RPC offloading
贡献指南
评估
这个 Issue 还没有评估数据。