abetlen / abetlen/llama-cpp-python

logprobs aren't returned if sampler will reach EOS

未关闭
#1,528 8 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
bug
主要语言
Python
星标
10.6k
派生
1.4k
PR 合并指标
PR 指标待抓取

描述

# Prerequisites

```python
llm = Llama(
model_path="/home/axyo/dev/LLM/models/Meta-Llama-3-8B-Instruct-GGUF-v2/Meta-Llama-3-8B-Instruct-v2.Q5_0.gguf",
n_gpu_layers=-1,
seed=8,
n_ctx=4096,
logits_all=True,
kv_overrides={"tokenizer.ggml.eos_token_id": 128002},
)

prompt = """<|start_header_id|>user<|end_header_id|>

What is a dog?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

A dog, also known as Canis lupus familiaris, is a domesticated mammal that belongs to the family Canidae. Dogs are closely related to wolves and share many physical and behavioral characteristics with them.

Dogs are typically characterized by their:

1. Physical appearance: Dogs come in a wide range of sizes, shapes, and coat types, from the tiny Chihuahua to the massive Great Dane. They have a furry coat, a wagging tail, and a distinctive snout.
2. Behavior: Dogs are social animals that thrive on interaction with their human family and other dogs. They are known for their loyalty, affection, and ability to form strong bonds with their owners.
3. Diet: Dogs are omnivores, which means they eat both plants and animals. They are often fed a diet of commercial dog food, which is formulated to meet their nutritional needs.
4. Communication: Dogs communicate primarily through body language and vocalizations, such as barking, whining, and growling.
5. Intelligence: Dogs are highly intelligent animals that are capable of learning a wide range of tasks and commands.

Dogs have been domesticated for thousands of years and have been bred for various purposes, such as:

* Companionship: Many dogs are kept as pets and are valued for their companionship and affection.
* Hunting: Some breeds of dogs, such as retrievers and pointers, are bred for hunting and retrieving game.
* Guarding: Some breeds, such as guard dogs and watchdogs, are bred for their protective instincts and ability to defend their territory.
* Assistance: Some dogs are trained as service animals to assist people with disabilities, such as guide dogs for the blind or hearing dogs for the deaf.

Overall, dogs are beloved animals that bring joy, companionship, and love to many people around the world

Would you like to know more about a specific aspect of dogs, such as their behavior, health, or training?"""

output = llm(
prompt,
echo=False,
logprobs=100,
max_tokens=1,

repeat_penalty=1.0, # disable penalties

top_k=1,
#top_p=1.0,
#min_p=-10000000000.0,
temperature=0,
)
print(output)
```

# Expected Behavior

logprobs!!

# Current Behavior

```
{'id': 'cmpl-c480f5df-6a62-4a39-9d21-d3058260523b', 'object': 'text_completion', 'created': 1718328460, 'model': '/home/axyo/dev/LLM/models/Meta-Llama-3-8B-Instruct-GGUF-v2/Meta-Llama-3-8B-Instruct-v2.Q5_0.gguf', 'choices': [{'text': '', 'index': 0, 'logprobs': {'tokens': [], 'text_offset': [], 'token_logprobs': [], 'top_logprobs': []}, 'finish_reason': 'stop'}], 'usage': {'prompt_tokens': 430, 'completion_tokens': 0, 'total_tokens': 430}}
```

# Environment and Context

llama_cpp_python-0.2.78

CUDA 12.4

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。