huggingface / huggingface/candle
how to use system prompt with the llama example?
- Dominant language
- Rust
- Stars
- 21k
- Forks
- 1.8k
- Avg merge
- 16h 42m
- Merged PRs (30d)
- 25
Description
Hi, I'm trying to pass a chat dialog in the [LLama3 format](https://github.com/meta-llama/llama3/blob/main/llama/tokenizer.py#L222) to the [llama example](https://github.com/huggingface/candle/tree/main/candle-examples/examples/llama) via -prompt, the string is as follows:
```
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful AI assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
Why is the sky blue?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
```
This seems to confuse the model and, depending on the user prompt can cause the model to generate gibberish characters (see also https://github.com/evilsocket/cake/issues/9):
(i've made a small change to load the prompt from a file if passed with @)
```sh
/path/to/compiled/llama3/example --model-id "meta-llama/Meta-Llama-3-8B" --prompt @hf-llama-test/prompt.txt
loading the model weights from meta-llama/Meta-Llama-3-8B
loading prompt from @hf-llama-test/prompt.txt ...
starting the inference loop
<|begin_of_text|><|start_header_id|>system<|end_header_id|>
You are a helpful AI assistant.<|eot_id|><|start_header_id|>user<|end_header_id|>
Why is the sky blue?<|eot_id|><|start_header_id|>assistant<|end_header_id|>
By: David Cope (2022, October 23)
14 tokens generated (16.831015425660595 token/s)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in candle-examples/examples/llama and trace how the --prompt argument is read, then reproduce the supplied Llama3-format prompt from hf-llama-test/prompt.txt. Compare the example behavior with the linked Llama tokenizer format; done means the expected system-prompt behavior is established and the reported gibberish is addressed or documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ai, cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100