huggingface / huggingface/lighteval
[BUG] Get low performance when evaluate Qwen3-8B on MMLU-Redux using the thinking mode.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
Hello,
I've recently been attempting to evaluate the `Qwen3-8B` model on the `MMLU-Redux` benchmark using `lighteval`.
To support the simultaneous switching between `thinking mode` and `no-thinking mode` , I made a minor modification to the `lighteval` source code: I added an `enable_thinking` argument during the`PromptManager` initialization. This allows the corresponding `enable_thinking` argument to be passed to `tokenizer.apply_chat_template()` when calling the internal `_prepare_chat_template()` function.
However, I believe this modification should be irrelevant to the evaluation of `MMLU-Redux` itself.
In the `no-thinking mode`, I achieved a **79.7** pass@1 result, which is very close to the **79.5** reported in the official Qwen3 technical report.
The command used was:
```
LIGHTEVAL_CONFIG="model_name=Qwen/Qwen3-8B,enable_thinking=false,dtype=bfloat16,max_model_length=40960,max_num_batched_tokens=32768,data_parallel_size=1,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.7,top_p:0.8,top_k:20}"
lighteval vllm \
${LIGHTEVAL_CONFIG} \
mmlu_redux_2 \
--remove-reasoning-tags
```
However, when I changed `enable_thinking` from `false` to `true` to evaluate the `thinking mode`, I only obtained a **51.8** pass@1 result. This is a massive drop in performance.
The results on several other benchmarks I've tested (such as `IFEval`, `GSM8K`, etc.) did not show a similar inverse contrast. Therefore, I suspect there might be an issue with the specific implementation of the `MMLU-Redux` evaluation within lighteval, possibly related to how the prompt or answer extraction interacts with the thought process.
Could you please look into this potential issue with the `MMLU-Redux` evaluation script?
Thank you for your time!
## To Reproduce
the default `enable_thinking` setting for Qwen3 tokenizer is true, so it can be ignored, and one can run evaluation on `MMLU-Redux` using `Qwen3-8B` `thinking mode` with the following cammand:
```
LIGHTEVAL_CONFIG="model_name=Qwen/Qwen3-8B,dtype=bfloat16,max_model_length=40960,max_num_batched_tokens=32768,data_parallel_size=1,gpu_memory_utilization=0.8,generation_parameters={max_new_tokens:32768,temperature:0.6,top_p:0.95,top_k:20}"
lighteval vllm \
${LIGHTEVAL_CONFIG} \
mmlu_redux_2 \
--remove-reasoning-tags
```
## Other Observation
Upon inspecting the evaluation logs, I noticed a large number of similar extraction fails:
```
We did not manage to extract a prediction in the correct format. Gold: ['C'], Pred: ["\n\nTo determine the correct conclusion from the given data, we need to apply the **Doppler effect** in the context of light. The Doppler effect describes how the wavelength of light from a moving source changes relative to an observer. This change in wavelength is what allows us to infer the motion of celestial objects.\n\n---\n\n### Key Concept: The Doppler Effect\n\n- **Blueshift**: When a source of light is moving **towards** the observer, the light waves are compressed, resulting in a **shorter wavelength** (blue shift).\n- **Redshift**: When a source of light is moving **away** from the observer, the light waves are stretched, resulting in a **longer wavelength** (red shift).\n\nIn this case:\n- The **rest wavelength** of the hydrogen spectral line is **486.1 nm**.\n- The **observed wavelength** in the star's spectrum is **485.9 nm**.\n\nThis means the observed wavelength is **shorter** than the rest wavelength, indicating a **blueshift**.\n\n---\n\n### Eliminating Other Options\n\n- **Option A: The star is getting hotter.** \n A change in the star's temperature would affect the **entire spectrum** (e.g., the blackbody curve), not just a specific spectral line. The position of a specific line (like the hydrogen line) is not directly related to the star's temperature. So this is **not the correct conclusion**.\n\n- **Option B: The star is getting colder.** \n Similar to option A, a change in temperature would not cause a specific spectral line to shift in wavelength. This is also **not the correct conclusion**.\n\n- **Option D: The star is moving away from us.** \n A redshift (longer wavelength) would indicate the star is moving away. Since the observed wavelength is **shorter**, this is **not the case**.\n\n- **Option C: The star is moving toward us.** \n A **blueshift** (shorter wavelength) is the result of the star moving **towards** the observer. This is **consistent** with the observed data.\n\n---\n\n### Final Answer\n\n$$\n\\boxed{C}\n$$"]
```
## Version info
To support `enable_thinking` , I just install `lighteval` from source and the version info show its `0.13.1.dev0`
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.