huggingface / huggingface/lighteval

Add a way to easily enable/disable thinking mode for reasoning models

Open
#855 2 comments 7 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Issue encountered
When evaluating reasoning capable models (e.g., Qwen3, etc) with lighteval, there's currently no way to toggle enable_thinking directly via CLI or config.

Is there a way to work around this or does it require a PR ?

## Solution/Feature
Maybe add a flag or config option to enable/disable thinking mode (i.e., pass enable_thinking=False to the tokenizer) for models that support it.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.