abetlen / abetlen/llama-cpp-python

Unclear and differing computation of conditional tokens' log probabilities

オープン
#1,276 コメント 0 件 リアクション 0 件 担当者 0 名 GitHub で見る
主要言語
Python
スター
10.6k
フォーク
1.4k
PR マージ指標
PR 指標を取得中

説明

Consider the following code snipped:

```python
from llama_cpp import Llama
import numpy as np

llm = Llama(
model_path=".mistral-7b-instruct-v0.2.Q4_K_M.gguf",
n_gpu_layers=-1,
logits_all=True
)

def eval_log_probs(llm, text):
# Unfortunately have to sample at least once:
output = llm(text, max_tokens=1, echo=True, logprobs=1)
# Remove last token:
tokens = output["choices"][0]["logprobs"]["tokens"][:-1]
# Remove last log_prob:
log_probs = output["choices"][0]["logprobs"]["token_logprobs"][:-1]
return tokens, log_probs

text = "[INST] What is your favourite condiment? [/INST]"

print(eval_log_probs(llm, text))
# '[', 'INST', ']', ' What', ' is', ' your', ' favourite', ' cond', 'iment', '?', ' [', '/', 'INST', ']'
# None, -11.821828, -11.579113, -13.468674, -0.6098522, -4.0169067, -4.8040905, -9.736423, -0.0067117964, -0.6900012, -10.652656, -9.419178, -11.562994, -12.2507105
```

In this model `` should get tokenized to the BOS token (my understanding is that the code relies on `llm.tokenize()` with `special=True`).
So it is not clear at this point why the conditional log prob of `[` given BOS is not reported (only the first log prob is not available from the model, as it would be the unconditional log prob of BOS), why is it the case?

To dig further, I looked at the code in `llama.py` and tried the following:

```python
tokens = llm.tokenize(text.encode("utf-8"), add_bos=False, special=True)
print(tokens)
# 1, 28792, 16289, 28793, 1824, 349, 574, 16020, 2076, 2487, 28804, 733, 28748, 16289, 28793
print([llm.detokenize([t]).decode("utf-8") for t in tokens])
# '', '[', 'INST', ']', ' What', ' is', ' your', ' favourite', ' cond', 'iment', '?', ' [', '/', 'INST', ']'

llm.reset()
llm.eval(tokens)
print(llm._scores.shape)
# (15, 32000)
print([Llama.logits_to_logprobs(llm._scores[i])[tokens[i+1]] for i in range(0, 14)])
# -19.646507, -12.055218, -10.872593, -13.957621, -0.60172236, -4.0697765, -4.9590707, -10.254766, -0.0028439811, -0.8368746, -10.785296, -10.573785, -11.5954895, -12.29845
```

I assumed that the `llm._scores` are the unnormalized log probs for the next token (logits), but the reported log probabilities differ, why is it the case?

Basically, I just want to compute the conditional log probabilities for all tokens in a given text (no sampling).

Version: llama_cpp_python==0.2.56.

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。