abetlen / abetlen/llama-cpp-python

Unclear and differing computation of conditional tokens' log probabilities

Đang mở
#1,276 0 bình luận 0 reaction 0 người được giao Xem trên GitHub
Ngôn ngữ chính
Python
Star
10.6k
Fork
1.4k
Chỉ số merge pull request
Chỉ số pull request đang chờ

Mô tả

Consider the following code snipped:

```python
from llama_cpp import Llama
import numpy as np

llm = Llama(
model_path=".mistral-7b-instruct-v0.2.Q4_K_M.gguf",
n_gpu_layers=-1,
logits_all=True
)

def eval_log_probs(llm, text):
# Unfortunately have to sample at least once:
output = llm(text, max_tokens=1, echo=True, logprobs=1)
# Remove last token:
tokens = output["choices"][0]["logprobs"]["tokens"][:-1]
# Remove last log_prob:
log_probs = output["choices"][0]["logprobs"]["token_logprobs"][:-1]
return tokens, log_probs

text = "[INST] What is your favourite condiment? [/INST]"

print(eval_log_probs(llm, text))
# '[', 'INST', ']', ' What', ' is', ' your', ' favourite', ' cond', 'iment', '?', ' [', '/', 'INST', ']'
# None, -11.821828, -11.579113, -13.468674, -0.6098522, -4.0169067, -4.8040905, -9.736423, -0.0067117964, -0.6900012, -10.652656, -9.419178, -11.562994, -12.2507105
```

In this model `` should get tokenized to the BOS token (my understanding is that the code relies on `llm.tokenize()` with `special=True`).
So it is not clear at this point why the conditional log prob of `[` given BOS is not reported (only the first log prob is not available from the model, as it would be the unconditional log prob of BOS), why is it the case?

To dig further, I looked at the code in `llama.py` and tried the following:

```python
tokens = llm.tokenize(text.encode("utf-8"), add_bos=False, special=True)
print(tokens)
# 1, 28792, 16289, 28793, 1824, 349, 574, 16020, 2076, 2487, 28804, 733, 28748, 16289, 28793
print([llm.detokenize([t]).decode("utf-8") for t in tokens])
# '', '[', 'INST', ']', ' What', ' is', ' your', ' favourite', ' cond', 'iment', '?', ' [', '/', 'INST', ']'

llm.reset()
llm.eval(tokens)
print(llm._scores.shape)
# (15, 32000)
print([Llama.logits_to_logprobs(llm._scores[i])[tokens[i+1]] for i in range(0, 14)])
# -19.646507, -12.055218, -10.872593, -13.957621, -0.60172236, -4.0697765, -4.9590707, -10.254766, -0.0028439811, -0.8368746, -10.785296, -10.573785, -11.5954895, -12.29845
```

I assumed that the `llm._scores` are the unnormalized log probs for the next token (logits), but the reported log probabilities differ, why is it the case?

Basically, I just want to compute the conditional log probabilities for all tokens in a given text (no sampling).

Version: llama_cpp_python==0.2.56.

Hướng dẫn đóng góp

Mở hướng dẫn đóng góp

Đánh giá

Issue này chưa được đánh giá.

Nhận issue mới trong hộp thư của bạn

Bản tóm tắt ngắn những issue GitHub phù hợp với người mới.