huggingface / huggingface/lighteval
lighteval eval crashes on fresh install: GenerateConfig rejects frequence_penalty / log_probs / response_format with current inspect-ai
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
**Summary.** A fresh `pip install lighteval` today resolves `inspect-ai` to the latest release (the dependency is declared as `inspect-ai>=0.3.140` with no upper bound), and `lighteval eval` then fails at startup for any model/task with a pydantic `ValidationError`: current `inspect_ai.model.GenerateConfig` rejects unknown fields.
**Repro:**
```
pip install lighteval # resolves inspect-ai 0.3.255 today
lighteval eval mockllm/model "lighteval|gsm8k|0" --max-samples 4 --log-dir out
```
```
ValidationError: 1 validation error for GenerateConfig
Value error, Unknown GenerateConfig field(s): frequence_penalty, log_probs, response_format.
```
**Root cause** (`lighteval/main_inspect.py`): the config is built with three field names current inspect-ai doesn't accept — `frequence_penalty` (lines 270/477 — note this one is a typo; inspect-ai spells it `frequency_penalty`), `log_probs` (line 483; inspect-ai: `logprobs`), and `response_format` (line 489; no longer a `GenerateConfig` field).
**Verified workaround** until a fix lands: `pip install "inspect-ai==0.3.140"` — with that pin the same command runs to completion and writes results.
**Suggested fix:** rename the two fields, route `response_format` through `extra_body` or drop it, and consider an upper bound on `inspect-ai` so releases can't break installs retroactively. Happy to open a small PR for the renames if useful.
Environment: Python 3.12.13, lighteval 0.13.0, inspect-ai 0.3.255 (broken) / 0.3.140 (works), macOS.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start in lighteval/main_inspect.py at the GenerateConfig construction around lines 270 and 477-489, then inspect the inspect-ai version constraint and its current GenerateConfig fields. Reproduce the fresh-install command with inspect-ai 0.3.255 and verify the supported handling of frequency_penalty, logprobs, and response_format. Done means the command completes on a fresh install and the relevant tests or evaluation run pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 75/100