huggingface / huggingface/lighteval
[BUG] System Prompt option now missing?
Open
bug
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
I was using an older version of LightEval packaged with OpenR1 repo, it had a `--system-prompt "xxx"` arg for lighteval.
That seems to be gone. I'm told that it is now a model arg, though I dont see that in the docs for vllm, but also if any comma is in the system prompt, it will break everything.
## To Reproduce
use
```
lighteval vllm "$MODEL_ARGS" "$benchmark_task" \
--system-prompt "$system_prompt" \
--use-chat-template \
--output-dir "$output_dir" 2>&1 | tee -a "$log_file"
```
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.