abetlen / abetlen/llama-cpp-python
llama_get_logits_ith: invalid logits id -1, reason: no logits
- Lingua principale
- Python
- Stelle
- 10.6k
- Fork
- 1.4k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
# llama_get_logits_ith: invalid logits id -1 error when embedding=True
## Expected Behavior
When using llama-cpp-python with Qwen2 model, the chat completion should work normally regardless of whether the embedding parameter is enabled or not.
## Current Behavior
The model works fine when `embedding=False`, but throws an error `llama_get_logits_ith: invalid logits id -1, reason: no logits` when `embedding=True`.
## Working Code Example
```python
from llama_cpp import Llama
# This works fine
llm = Llama(
model_path="./models/qwen2-0_5b-instruct-q8_0.gguf",
chat_format="chatml",
verbose=False
)
messages = [
{"role": "system", "content": "Summarize this text for me: You are an assistant who creates short stories."},
{"role": "user", "content": "Long ago, in a peaceful village, a little girl named Leah loved watching the stars at night..."}
]
response = llm.create_chat_completion(messages=messages)
'''
{'id': 'chatcmpl-17ca45ef-d13b-425a-96be-7631e3b9a7f4',
'object': 'chat.completion',
'created': 1730125699,
'model': './models/qwen2-0_5b-instruct-q8_0.gguf',
'choices': [{'index': 0,
'message': {'role': 'assistant',
'content': 'This text is a short story about a little girl named Leah who loves watching the stars at night. One day, she noticed a particularly bright star that seemed to wink at her, and she made a wish to become friends with the star. This star spirit helped Leah take her on a magical adventure among the stars, and she visited countless constellations and stardust rivers.'},
'logprobs': None,
'finish_reason': 'stop'}],
'usage': {'prompt_tokens': 145, 'completion_tokens': 76, 'total_tokens': 221}
}
'''
# Works successfully
```
## Error Reproduction
```python
from llama_cpp import Llama
# This causes an error
llm = Llama(
model_path="./models/qwen2-0_5b-instruct-q8_0.gguf",
chat_format="chatml",
verbose=False,
embedding=True # Only difference is enabling embedding
)
messages = [
{"role": "system", "content": "Summarize this text for me: You are an assistant who creates short stories."},
{"role": "user", "content": "Long ago, in a peaceful village, a little girl named Leah loved watching the stars at night..."}
]
llm.create_chat_completion(messages=messages)
# Error: llama_get_logits_ith: invalid logits id -1, reason: no logits
embeddings = llm.create_embedding("Hello, world!")
# Here is normal
'''
{'object': 'list',
'data': [{'object': 'embedding',
'embedding': [[0.9160200953483582,
5.090432167053223,
1.487088680267334, ......
'''
```
## Environment Info
- Python version: 3.10
- llama-cpp-python version: latest
- Model: Qwen2-0.5B-Chat (GGUF format)
## Steps to Reproduce
1. Install llama-cpp-python
2. Download Qwen2-0.5B-Chat GGUF model
3. Run the error reproduction code above with `embedding=True`
## Additional Context
The error only occurs when:
1. The `embedding` parameter is set to `True`
2. Using the chat completion functionality
The model works fine for chat completion when `embedding=False`, suggesting this might be related to how the embedding functionality is implemented for this specific model.
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.