huggingface / huggingface/lighteval
Add a way to easily enable/disable thinking mode for reasoning models
Open
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Issue encountered
When evaluating reasoning capable models (e.g., Qwen3, etc) with lighteval, there's currently no way to toggle enable_thinking directly via CLI or config.
Is there a way to work around this or does it require a PR ?
## Solution/Feature
Maybe add a flag or config option to enable/disable thinking mode (i.e., pass enable_thinking=False to the tokenizer) for models that support it.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.