huggingface / huggingface/lighteval

[BUG] Logs are missing some fields, truncation likely buggy

Open
#892 2 comments 0 reactions 1 assignee Claimed by @NathanHB View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug
With a model max length at 10K on ifeval, I'm getting truncations on the input (with no input stored in the logs): most likely an issue with the truncation.

I suspect it's also linked to the refacto on loglikelihood: the padding no longer works for accelerate models.

## To Reproduce
`lighteval accelerate 'model_name=Qwen/Qwen3-32B,model_parallel=True,max_length=16384,generation_parameters={temperature:0.6,top_p:0.9,top_k:40,min_p:0,max_new_tokens:16384}' 'extended|ifeval|0|0' --save-details --output-dir tmp`

Logs will contain empty input fields in ModelResponse, with around 400 truncation

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.