huggingface / huggingface/lighteval

lighteval eval crashes on fresh install: GenerateConfig rejects frequence_penalty / log_probs / response_format with current inspect-ai

Open
#1,327 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

**Summary.** A fresh `pip install lighteval` today resolves `inspect-ai` to the latest release (the dependency is declared as `inspect-ai>=0.3.140` with no upper bound), and `lighteval eval` then fails at startup for any model/task with a pydantic `ValidationError`: current `inspect_ai.model.GenerateConfig` rejects unknown fields.

**Repro:**
```
pip install lighteval # resolves inspect-ai 0.3.255 today
lighteval eval mockllm/model "lighteval|gsm8k|0" --max-samples 4 --log-dir out
```
```
ValidationError: 1 validation error for GenerateConfig
Value error, Unknown GenerateConfig field(s): frequence_penalty, log_probs, response_format.
```

**Root cause** (`lighteval/main_inspect.py`): the config is built with three field names current inspect-ai doesn't accept — `frequence_penalty` (lines 270/477 — note this one is a typo; inspect-ai spells it `frequency_penalty`), `log_probs` (line 483; inspect-ai: `logprobs`), and `response_format` (line 489; no longer a `GenerateConfig` field).

**Verified workaround** until a fix lands: `pip install "inspect-ai==0.3.140"` — with that pin the same command runs to completion and writes results.

**Suggested fix:** rename the two fields, route `response_format` through `extra_body` or drop it, and consider an upper bound on `inspect-ai` so releases can't break installs retroactively. Happy to open a small PR for the renames if useful.

Environment: Python 3.12.13, lighteval 0.13.0, inspect-ai 0.3.255 (broken) / 0.3.140 (works), macOS.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in lighteval/main_inspect.py at the GenerateConfig construction around lines 270 and 477-489, then inspect the inspect-ai version constraint and its current GenerateConfig fields. Reproduce the fresh-install command with inspect-ai 0.3.255 and verify the supported handling of frequency_penalty, logprobs, and response_format. Done means the command completes on a fresh install and the relevant tests or evaluation run pass.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
75/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.