abetlen / abetlen/llama-cpp-python

Unclear and differing computation of conditional tokens' log probabilities

Ouverte
#1,276 0 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Python
Étoiles
10.6k
Forks
1.4k
Métriques de merge des PR
Métriques de PR en attente

Description

Consider the following code snipped:

```python
from llama_cpp import Llama
import numpy as np

llm = Llama(
model_path=".mistral-7b-instruct-v0.2.Q4_K_M.gguf",
n_gpu_layers=-1,
logits_all=True
)

def eval_log_probs(llm, text):
# Unfortunately have to sample at least once:
output = llm(text, max_tokens=1, echo=True, logprobs=1)
# Remove last token:
tokens = output["choices"][0]["logprobs"]["tokens"][:-1]
# Remove last log_prob:
log_probs = output["choices"][0]["logprobs"]["token_logprobs"][:-1]
return tokens, log_probs

text = "[INST] What is your favourite condiment? [/INST]"

print(eval_log_probs(llm, text))
# '[', 'INST', ']', ' What', ' is', ' your', ' favourite', ' cond', 'iment', '?', ' [', '/', 'INST', ']'
# None, -11.821828, -11.579113, -13.468674, -0.6098522, -4.0169067, -4.8040905, -9.736423, -0.0067117964, -0.6900012, -10.652656, -9.419178, -11.562994, -12.2507105
```

In this model `` should get tokenized to the BOS token (my understanding is that the code relies on `llm.tokenize()` with `special=True`).
So it is not clear at this point why the conditional log prob of `[` given BOS is not reported (only the first log prob is not available from the model, as it would be the unconditional log prob of BOS), why is it the case?

To dig further, I looked at the code in `llama.py` and tried the following:

```python
tokens = llm.tokenize(text.encode("utf-8"), add_bos=False, special=True)
print(tokens)
# 1, 28792, 16289, 28793, 1824, 349, 574, 16020, 2076, 2487, 28804, 733, 28748, 16289, 28793
print([llm.detokenize([t]).decode("utf-8") for t in tokens])
# '', '[', 'INST', ']', ' What', ' is', ' your', ' favourite', ' cond', 'iment', '?', ' [', '/', 'INST', ']'

llm.reset()
llm.eval(tokens)
print(llm._scores.shape)
# (15, 32000)
print([Llama.logits_to_logprobs(llm._scores[i])[tokens[i+1]] for i in range(0, 14)])
# -19.646507, -12.055218, -10.872593, -13.957621, -0.60172236, -4.0697765, -4.9590707, -10.254766, -0.0028439811, -0.8368746, -10.785296, -10.573785, -11.5954895, -12.29845
```

I assumed that the `llm._scores` are the unnormalized log probs for the next token (logits), but the reported log probabilities differ, why is it the case?

Basically, I just want to compute the conditional log probabilities for all tokens in a given text (no sampling).

Version: llama_cpp_python==0.2.56.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.