abetlen / abetlen/llama-cpp-python

llama_get_logits_ith: invalid logits id -1, reason: no logits

Ouverte
#1,812 3 commentaires 0 réactions 0 personnes assignées Voir sur GitHub
Langage dominant
Python
Étoiles
10.6k
Forks
1.4k
Métriques de merge des PR
Métriques de PR en attente

Description

# llama_get_logits_ith: invalid logits id -1 error when embedding=True

## Expected Behavior

When using llama-cpp-python with Qwen2 model, the chat completion should work normally regardless of whether the embedding parameter is enabled or not.

## Current Behavior

The model works fine when `embedding=False`, but throws an error `llama_get_logits_ith: invalid logits id -1, reason: no logits` when `embedding=True`.

## Working Code Example

```python
from llama_cpp import Llama

# This works fine
llm = Llama(
model_path="./models/qwen2-0_5b-instruct-q8_0.gguf",
chat_format="chatml",
verbose=False
)

messages = [
{"role": "system", "content": "Summarize this text for me: You are an assistant who creates short stories."},
{"role": "user", "content": "Long ago, in a peaceful village, a little girl named Leah loved watching the stars at night..."}
]

response = llm.create_chat_completion(messages=messages)

'''
{'id': 'chatcmpl-17ca45ef-d13b-425a-96be-7631e3b9a7f4',
'object': 'chat.completion',
'created': 1730125699,
'model': './models/qwen2-0_5b-instruct-q8_0.gguf',
'choices': [{'index': 0,
'message': {'role': 'assistant',
'content': 'This text is a short story about a little girl named Leah who loves watching the stars at night. One day, she noticed a particularly bright star that seemed to wink at her, and she made a wish to become friends with the star. This star spirit helped Leah take her on a magical adventure among the stars, and she visited countless constellations and stardust rivers.'},
'logprobs': None,
'finish_reason': 'stop'}],
'usage': {'prompt_tokens': 145, 'completion_tokens': 76, 'total_tokens': 221}
}
'''

# Works successfully
```

## Error Reproduction

```python
from llama_cpp import Llama

# This causes an error
llm = Llama(
model_path="./models/qwen2-0_5b-instruct-q8_0.gguf",
chat_format="chatml",
verbose=False,
embedding=True # Only difference is enabling embedding
)

messages = [
{"role": "system", "content": "Summarize this text for me: You are an assistant who creates short stories."},
{"role": "user", "content": "Long ago, in a peaceful village, a little girl named Leah loved watching the stars at night..."}
]

llm.create_chat_completion(messages=messages)
# Error: llama_get_logits_ith: invalid logits id -1, reason: no logits

embeddings = llm.create_embedding("Hello, world!")
# Here is normal

'''
{'object': 'list',
'data': [{'object': 'embedding',
'embedding': [[0.9160200953483582,
5.090432167053223,
1.487088680267334, ......
'''
```

## Environment Info

- Python version: 3.10
- llama-cpp-python version: latest
- Model: Qwen2-0.5B-Chat (GGUF format)

## Steps to Reproduce

1. Install llama-cpp-python
2. Download Qwen2-0.5B-Chat GGUF model
3. Run the error reproduction code above with `embedding=True`

## Additional Context

The error only occurs when:
1. The `embedding` parameter is set to `True`
2. Using the chat completion functionality

The model works fine for chat completion when `embedding=False`, suggesting this might be related to how the embedding functionality is implemented for this specific model.

Guide de contribution

Ouvrir le guide de contribution

Évaluation

Cette issue n'a pas encore été évaluée.

Recevez les nouvelles issues par e-mail

Un résumé court des issues GitHub adaptées aux débutants.