huggingface / huggingface/lighteval
[BUG] Sampling and max new tokens params for accelerate backend not being applied correctly
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
In regards to below lighteval command using `accelerate` backend, the generation params for sampling and max new tokens are not being applied correctly.
Observing the following warning (moreover, setting do_sample:true throws a transformer config validation error)
```2025-05-19 10:08:27,393] [ WARNING]: /usr/local/lib/python3.11/dist-packages/transformers/generation/configuration_utils.py:631: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.5` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
warnings.warn(
(warnings.py:110)```
## To Reproduce
Please provide all the steps needed to reproduce the behavior, or provide a minimal working example if needed. We will ignore issues missing this section.
Command:
`lighteval accelerate "model_name=Qwen/Qwen3-1.7B,generation_parameters={temperature:0.6, top_k:20, top_p:0.95, max_new_tokens:32768}" "lighteval|gsm8k|0|0" --system-prompt "You are an expert in math. Provide the final numeric answer in the following format: #### 123" --save-details --use-chat-template`
## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.
OS - Ubuntu 22.04.5 LTS
Python - 3.11.10
pip - 24.2
lighteval - 0.9.2
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.