huggingface / huggingface/candle

how to use system prompt with the llama example?

Open
#2,341 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
21k
Forks
1.8k
Avg merge
16h 42m
Merged PRs (30d)
25

Description

Hi, I'm trying to pass a chat dialog in the [LLama3 format](https://github.com/meta-llama/llama3/blob/main/llama/tokenizer.py#L222) to the [llama example](https://github.com/huggingface/candle/tree/main/candle-examples/examples/llama) via -prompt, the string is as follows:

```
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful AI assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

Why is the sky blue?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

```

This seems to confuse the model and, depending on the user prompt can cause the model to generate gibberish characters (see also https://github.com/evilsocket/cake/issues/9):

(i've made a small change to load the prompt from a file if passed with @)

```sh
/path/to/compiled/llama3/example --model-id "meta-llama/Meta-Llama-3-8B" --prompt @hf-llama-test/prompt.txt

loading the model weights from meta-llama/Meta-Llama-3-8B
loading prompt from @hf-llama-test/prompt.txt ...
starting the inference loop
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a helpful AI assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>

Why is the sky blue?<|eot_id|><|start_header_id|>assistant<|end_header_id|>

By: David Cope (2022, October 23)

14 tokens generated (16.831015425660595 token/s)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in candle-examples/examples/llama and trace how the --prompt argument is read, then reproduce the supplied Llama3-format prompt from hf-llama-test/prompt.txt. Compare the example behavior with the linked Llama tokenizer format; done means the expected system-prompt behavior is established and the reported gibberish is addressed or documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
ai, cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.