abetlen / abetlen/llama-cpp-python

llama_get_logits_ith: invalid logits id -1, reason: no logits

Open
#1,812 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
10.6k
Forks
1.4k
PR merge metrics
PR metrics pending

Description

# llama_get_logits_ith: invalid logits id -1 error when embedding=True

## Expected Behavior

When using llama-cpp-python with Qwen2 model, the chat completion should work normally regardless of whether the embedding parameter is enabled or not.

## Current Behavior

The model works fine when `embedding=False`, but throws an error `llama_get_logits_ith: invalid logits id -1, reason: no logits` when `embedding=True`.

## Working Code Example

```python
from llama_cpp import Llama

# This works fine
llm = Llama(
model_path="./models/qwen2-0_5b-instruct-q8_0.gguf",
chat_format="chatml",
verbose=False
)

messages = [
{"role": "system", "content": "Summarize this text for me: You are an assistant who creates short stories."},
{"role": "user", "content": "Long ago, in a peaceful village, a little girl named Leah loved watching the stars at night..."}
]

response = llm.create_chat_completion(messages=messages)

'''
{'id': 'chatcmpl-17ca45ef-d13b-425a-96be-7631e3b9a7f4',
'object': 'chat.completion',
'created': 1730125699,
'model': './models/qwen2-0_5b-instruct-q8_0.gguf',
'choices': [{'index': 0,
'message': {'role': 'assistant',
'content': 'This text is a short story about a little girl named Leah who loves watching the stars at night. One day, she noticed a particularly bright star that seemed to wink at her, and she made a wish to become friends with the star. This star spirit helped Leah take her on a magical adventure among the stars, and she visited countless constellations and stardust rivers.'},
'logprobs': None,
'finish_reason': 'stop'}],
'usage': {'prompt_tokens': 145, 'completion_tokens': 76, 'total_tokens': 221}
}
'''

# Works successfully
```

## Error Reproduction

```python
from llama_cpp import Llama

# This causes an error
llm = Llama(
model_path="./models/qwen2-0_5b-instruct-q8_0.gguf",
chat_format="chatml",
verbose=False,
embedding=True # Only difference is enabling embedding
)

messages = [
{"role": "system", "content": "Summarize this text for me: You are an assistant who creates short stories."},
{"role": "user", "content": "Long ago, in a peaceful village, a little girl named Leah loved watching the stars at night..."}
]

llm.create_chat_completion(messages=messages)
# Error: llama_get_logits_ith: invalid logits id -1, reason: no logits

embeddings = llm.create_embedding("Hello, world!")
# Here is normal

'''
{'object': 'list',
'data': [{'object': 'embedding',
'embedding': [[0.9160200953483582,
5.090432167053223,
1.487088680267334, ......
'''
```

## Environment Info

- Python version: 3.10
- llama-cpp-python version: latest
- Model: Qwen2-0.5B-Chat (GGUF format)

## Steps to Reproduce

1. Install llama-cpp-python
2. Download Qwen2-0.5B-Chat GGUF model
3. Run the error reproduction code above with `embedding=True`

## Additional Context

The error only occurs when:
1. The `embedding` parameter is set to `True`
2. Using the chat completion functionality

The model works fine for chat completion when `embedding=False`, suggesting this might be related to how the embedding functionality is implemented for this specific model.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.