abetlen / abetlen/llama-cpp-python

Unclear and differing computation of conditional tokens' log probabilities

Offen
#1,276 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
Vorherrschende Sprache
Python
Sterne
10.6k
Forks
1.4k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

Consider the following code snipped:

```python
from llama_cpp import Llama
import numpy as np

llm = Llama(
model_path=".mistral-7b-instruct-v0.2.Q4_K_M.gguf",
n_gpu_layers=-1,
logits_all=True
)

def eval_log_probs(llm, text):
# Unfortunately have to sample at least once:
output = llm(text, max_tokens=1, echo=True, logprobs=1)
# Remove last token:
tokens = output["choices"][0]["logprobs"]["tokens"][:-1]
# Remove last log_prob:
log_probs = output["choices"][0]["logprobs"]["token_logprobs"][:-1]
return tokens, log_probs

text = "[INST] What is your favourite condiment? [/INST]"

print(eval_log_probs(llm, text))
# '[', 'INST', ']', ' What', ' is', ' your', ' favourite', ' cond', 'iment', '?', ' [', '/', 'INST', ']'
# None, -11.821828, -11.579113, -13.468674, -0.6098522, -4.0169067, -4.8040905, -9.736423, -0.0067117964, -0.6900012, -10.652656, -9.419178, -11.562994, -12.2507105
```

In this model `` should get tokenized to the BOS token (my understanding is that the code relies on `llm.tokenize()` with `special=True`).
So it is not clear at this point why the conditional log prob of `[` given BOS is not reported (only the first log prob is not available from the model, as it would be the unconditional log prob of BOS), why is it the case?

To dig further, I looked at the code in `llama.py` and tried the following:

```python
tokens = llm.tokenize(text.encode("utf-8"), add_bos=False, special=True)
print(tokens)
# 1, 28792, 16289, 28793, 1824, 349, 574, 16020, 2076, 2487, 28804, 733, 28748, 16289, 28793
print([llm.detokenize([t]).decode("utf-8") for t in tokens])
# '', '[', 'INST', ']', ' What', ' is', ' your', ' favourite', ' cond', 'iment', '?', ' [', '/', 'INST', ']'

llm.reset()
llm.eval(tokens)
print(llm._scores.shape)
# (15, 32000)
print([Llama.logits_to_logprobs(llm._scores[i])[tokens[i+1]] for i in range(0, 14)])
# -19.646507, -12.055218, -10.872593, -13.957621, -0.60172236, -4.0697765, -4.9590707, -10.254766, -0.0028439811, -0.8368746, -10.785296, -10.573785, -11.5954895, -12.29845
```

I assumed that the `llm._scores` are the unnormalized log probs for the next token (logits), but the reported log probabilities differ, why is it the case?

Basically, I just want to compute the conditional log probabilities for all tokens in a given text (no sampling).

Version: llama_cpp_python==0.2.56.

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.