abetlen / abetlen/llama-cpp-python
A different value of n_batch gives different logits (llama.cpp_hf)
- Ngôn ngữ chính
- Python
- Star
- 10.6k
- Fork
- 1.4k
- Chỉ số merge pull request
- Chỉ số pull request đang chờ
Mô tả
Hello,
I was told that n_batch doesn't alter the logits but after doing some tests I can see that it's not working as intented (or is it?).
"*When was born Tony Yoka? - Tony Yoka was born the 28 April 199...*"
The next token is supposed to be "2"

You can see that a low n_batch value objectively leads to a deterioration in the quality of the results (The only viable token is less likely to be chosen).
### Is there an existing issue for this?
- [X] I have searched the existing issues
### Reproduction
Change the n_batch value and compare the logits.
### Screenshot
_No response_
### Logs
```shell
No logs.
```
### System Info
```shell
Windows 10.
RTX 3060.
```
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.