abetlen / abetlen/llama-cpp-python

Confusion regarding operation/terminology of speculative decoding and sampling

オープン
#1,865 コメント 2 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
10.6k
フォーク
1.4k
PR マージ指標
PR 指標を取得中

説明

# Summary

Speculative decoding not interfacing as expected.

My understanding is that a draft model should be specified, and that the draft model should itself be another llm (typically, smaller with similar/same token formatting). This understanding is derived from https://github.com/ggerganov/llama.cpp/pull/2926 which is referenced in some llama-cpp-python issues, specifically https://github.com/abetlen/llama-cpp-python/issues/675 . I've also come across https://github.com/abetlen/llama-cpp-python/pull/1120 .

There does not seem to be a direct interface to specify another llm, yet there is a "draft_model" argument in Llama() which instead points to a LlamaPromptLookupDecoding object. This is where my confusion arises from.

# Expected Behavior

Adjusting the current example on the main page, i would have expected speculative decoding to operate something like the following:

```
from llama_cpp import Llama

llama_draft = (
model_path="path/to/**small_draft_model.gguf**"
)

llama = Llama(
model_path="path/to/**big_primary_model**.gguf",
draft_model=llama_draft
)
```

# Current Behavior

this is the current suggested operation:

```
from llama_cpp import Llama
from llama_cpp.llama_speculative import LlamaPromptLookupDecoding

llama = Llama(
model_path="path/to/model.gguf",
draft_model=LlamaPromptLookupDecoding(num_pred_tokens=10) # num_pred_tokens is the number of tokens to predict 10 is the default and generally good for gpu, 2 performs better for cpu-only machines.
)
```

here, the draft_model is not another llm.

Just asking for some clarification and if it's currently possible or on the roadmap to implement the expected behaviour

cheers!

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。