abetlen / abetlen/llama-cpp-python

A different value of n_batch gives different logits (llama.cpp_hf)

Offen
#1,018 0 Kommentare 0 Reaktionen 0 zugewiesene Personen Auf GitHub ansehen
bug
Vorherrschende Sprache
Python
Sterne
10.6k
Forks
1.4k
PR-Merge-Kennzahlen
PR-Kennzahlen ausstehend

Beschreibung

Hello,

I was told that n_batch doesn't alter the logits but after doing some tests I can see that it's not working as intented (or is it?).

"*When was born Tony Yoka? - Tony Yoka was born the 28 April 199...*"
The next token is supposed to be "2"
![N8IiAaQ](https://github.com/oobabooga/text-generation-webui/assets/110173477/b854ad25-c5a6-4a3c-a404-832ee14a43c3)

You can see that a low n_batch value objectively leads to a deterioration in the quality of the results (The only viable token is less likely to be chosen).

### Is there an existing issue for this?

- [X] I have searched the existing issues

### Reproduction

Change the n_batch value and compare the logits.

### Screenshot

_No response_

### Logs

```shell
No logs.
```

### System Info

```shell
Windows 10.
RTX 3060.
```

Beitragsleitfaden

Beitragsleitfaden öffnen

Bewertung

Dieses Issue wurde noch nicht bewertet.

Neue Issues direkt in Ihr Postfach

Eine kurze Übersicht über anfängerfreundliche GitHub-Issues.