abetlen / abetlen/llama-cpp-python

RPC is broken due to change of interface in llama.cpp main repository (rpc : early register backend devices #11262)

未关闭
#1,914 5 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
主要语言
Python
星标
10.6k
派生
1.4k
PR 合并指标
PR 指标待抓取

描述

RPC does not work in llama-cpp-python since this commit in [llama.cpp](https://github.com/ggerganov/llama.cpp)
```
Revision: 667d72846c06b2cf4f7c8a4265e210991a49706b
Author: Radoslav Gerganov
Date: 17/01/2025 09:57:09
Message:
rpc : early register backend devices (#11262)
Early register RPC devices and do not propagate RPC specifics in the
llama model structures.

This commit change the way rpc_servers are passed to llama.cpp, (in llama.cpp\common\arg.cpp) it is not a parameter anymore (params.rpc_servers = value;) but need to call add_rpc_devices(value);
```

To reproduce:
```
from llama_cpp import Llama
llm = Llama(
model_path=MODEL,
rpc_servers=RPC_NODES,
n_ctx = 4096,
flash_attn = False,
n_gpu_layers = 99,
verbose=True
)
```
--> should init with RPC offloading

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。