huggingface / huggingface/lighteval

[BUG] System Prompt option now missing?

Open
#844 4 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
I was using an older version of LightEval packaged with OpenR1 repo, it had a `--system-prompt "xxx"` arg for lighteval.

That seems to be gone. I'm told that it is now a model arg, though I dont see that in the docs for vllm, but also if any comma is in the system prompt, it will break everything.

## To Reproduce
use

```
lighteval vllm "$MODEL_ARGS" "$benchmark_task" \
--system-prompt "$system_prompt" \
--use-chat-template \
--output-dir "$output_dir" 2>&1 | tee -a "$log_file"
```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.