huggingface / huggingface/lighteval

[BUG] Sampling and max new tokens params for accelerate backend not being applied correctly

Open
#742 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
In regards to below lighteval command using `accelerate` backend, the generation params for sampling and max new tokens are not being applied correctly.

Observing the following warning (moreover, setting do_sample:true throws a transformer config validation error)

```2025-05-19 10:08:27,393] [ WARNING]: /usr/local/lib/python3.11/dist-packages/transformers/generation/configuration_utils.py:631: UserWarning: `do_sample` is set to `False`. However, `temperature` is set to `0.5` -- this flag is only used in sample-based generation modes. You should set `do_sample=True` or unset `temperature`.
warnings.warn(
(warnings.py:110)```

## To Reproduce
Please provide all the steps needed to reproduce the behavior, or provide a minimal working example if needed. We will ignore issues missing this section.

Command:
`lighteval accelerate "model_name=Qwen/Qwen3-1.7B,generation_parameters={temperature:0.6, top_k:20, top_p:0.95, max_new_tokens:32768}" "lighteval|gsm8k|0|0" --system-prompt "You are an expert in math. Provide the final numeric answer in the following format: #### 123" --save-details --use-chat-template`

## Version info
Please provide your operating system, lighteval version or commit if you installed from main, and pip/conda environment if your problem concerns dependencies.

OS - Ubuntu 22.04.5 LTS
Python - 3.11.10
pip - 24.2
lighteval - 0.9.2

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.