huggingface / huggingface/lighteval
[BUG] Logs are missing some fields, truncation likely buggy
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
With a model max length at 10K on ifeval, I'm getting truncations on the input (with no input stored in the logs): most likely an issue with the truncation.
I suspect it's also linked to the refacto on loglikelihood: the padding no longer works for accelerate models.
## To Reproduce
`lighteval accelerate 'model_name=Qwen/Qwen3-32B,model_parallel=True,max_length=16384,generation_parameters={temperature:0.6,top_p:0.9,top_k:40,min_p:0,max_new_tokens:16384}' 'extended|ifeval|0|0' --save-details --output-dir tmp`
Logs will contain empty input fields in ModelResponse, with around 400 truncation
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.