abetlen / abetlen/llama-cpp-python
Unclear and differing computation of conditional tokens' log probabilities
- 主要语言
- Python
- 星标
- 10.6k
- 派生
- 1.4k
- PR 合并指标
- PR 指标待抓取
描述
Consider the following code snipped:
```python
from llama_cpp import Llama
import numpy as np
llm = Llama(
model_path=".mistral-7b-instruct-v0.2.Q4_K_M.gguf",
n_gpu_layers=-1,
logits_all=True
)
def eval_log_probs(llm, text):
# Unfortunately have to sample at least once:
output = llm(text, max_tokens=1, echo=True, logprobs=1)
# Remove last token:
tokens = output["choices"][0]["logprobs"]["tokens"][:-1]
# Remove last log_prob:
log_probs = output["choices"][0]["logprobs"]["token_logprobs"][:-1]
return tokens, log_probs
text = "[INST] What is your favourite condiment? [/INST]"
print(eval_log_probs(llm, text))
# '[', 'INST', ']', ' What', ' is', ' your', ' favourite', ' cond', 'iment', '?', ' [', '/', 'INST', ']'
# None, -11.821828, -11.579113, -13.468674, -0.6098522, -4.0169067, -4.8040905, -9.736423, -0.0067117964, -0.6900012, -10.652656, -9.419178, -11.562994, -12.2507105
```
In this model `` should get tokenized to the BOS token (my understanding is that the code relies on `llm.tokenize()` with `special=True`).
So it is not clear at this point why the conditional log prob of `[` given BOS is not reported (only the first log prob is not available from the model, as it would be the unconditional log prob of BOS), why is it the case?
To dig further, I looked at the code in `llama.py` and tried the following:
```python
tokens = llm.tokenize(text.encode("utf-8"), add_bos=False, special=True)
print(tokens)
# 1, 28792, 16289, 28793, 1824, 349, 574, 16020, 2076, 2487, 28804, 733, 28748, 16289, 28793
print([llm.detokenize([t]).decode("utf-8") for t in tokens])
# '', '[', 'INST', ']', ' What', ' is', ' your', ' favourite', ' cond', 'iment', '?', ' [', '/', 'INST', ']'
llm.reset()
llm.eval(tokens)
print(llm._scores.shape)
# (15, 32000)
print([Llama.logits_to_logprobs(llm._scores[i])[tokens[i+1]] for i in range(0, 14)])
# -19.646507, -12.055218, -10.872593, -13.957621, -0.60172236, -4.0697765, -4.9590707, -10.254766, -0.0028439811, -0.8368746, -10.785296, -10.573785, -11.5954895, -12.29845
```
I assumed that the `llm._scores` are the unnormalized log probs for the next token (logits), but the reported log probabilities differ, why is it the case?
Basically, I just want to compute the conditional log probabilities for all tokens in a given text (no sampling).
Version: llama_cpp_python==0.2.56.
贡献指南
评估
这个 Issue 还没有评估数据。