abetlen / abetlen/llama-cpp-python

llama_get_logits_ith: invalid logits id -1, reason: no logits

Aperta
#1,812 3 commenti 0 reazioni 0 assegnatari Vedi su GitHub
Lingua principale
Python
Stelle
10.6k
Fork
1.4k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

# llama_get_logits_ith: invalid logits id -1 error when embedding=True

## Expected Behavior

When using llama-cpp-python with Qwen2 model, the chat completion should work normally regardless of whether the embedding parameter is enabled or not.

## Current Behavior

The model works fine when `embedding=False`, but throws an error `llama_get_logits_ith: invalid logits id -1, reason: no logits` when `embedding=True`.

## Working Code Example

```python
from llama_cpp import Llama

# This works fine
llm = Llama(
model_path="./models/qwen2-0_5b-instruct-q8_0.gguf",
chat_format="chatml",
verbose=False
)

messages = [
{"role": "system", "content": "Summarize this text for me: You are an assistant who creates short stories."},
{"role": "user", "content": "Long ago, in a peaceful village, a little girl named Leah loved watching the stars at night..."}
]

response = llm.create_chat_completion(messages=messages)

'''
{'id': 'chatcmpl-17ca45ef-d13b-425a-96be-7631e3b9a7f4',
'object': 'chat.completion',
'created': 1730125699,
'model': './models/qwen2-0_5b-instruct-q8_0.gguf',
'choices': [{'index': 0,
'message': {'role': 'assistant',
'content': 'This text is a short story about a little girl named Leah who loves watching the stars at night. One day, she noticed a particularly bright star that seemed to wink at her, and she made a wish to become friends with the star. This star spirit helped Leah take her on a magical adventure among the stars, and she visited countless constellations and stardust rivers.'},
'logprobs': None,
'finish_reason': 'stop'}],
'usage': {'prompt_tokens': 145, 'completion_tokens': 76, 'total_tokens': 221}
}
'''

# Works successfully
```

## Error Reproduction

```python
from llama_cpp import Llama

# This causes an error
llm = Llama(
model_path="./models/qwen2-0_5b-instruct-q8_0.gguf",
chat_format="chatml",
verbose=False,
embedding=True # Only difference is enabling embedding
)

messages = [
{"role": "system", "content": "Summarize this text for me: You are an assistant who creates short stories."},
{"role": "user", "content": "Long ago, in a peaceful village, a little girl named Leah loved watching the stars at night..."}
]

llm.create_chat_completion(messages=messages)
# Error: llama_get_logits_ith: invalid logits id -1, reason: no logits

embeddings = llm.create_embedding("Hello, world!")
# Here is normal

'''
{'object': 'list',
'data': [{'object': 'embedding',
'embedding': [[0.9160200953483582,
5.090432167053223,
1.487088680267334, ......
'''
```

## Environment Info

- Python version: 3.10
- llama-cpp-python version: latest
- Model: Qwen2-0.5B-Chat (GGUF format)

## Steps to Reproduce

1. Install llama-cpp-python
2. Download Qwen2-0.5B-Chat GGUF model
3. Run the error reproduction code above with `embedding=True`

## Additional Context

The error only occurs when:
1. The `embedding` parameter is set to `True`
2. Using the chat completion functionality

The model works fine for chat completion when `embedding=False`, suggesting this might be related to how the embedding functionality is implemented for this specific model.

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.