huggingface / huggingface/lighteval
[BUG] dyck_language scores with inspect_ai's exact(), which ignored word order in every release up to 0.3.252
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 555
- Avg merge
- 1d 6h
- Merged PRs (30d)
- 1
Description
## Describe the bug
`src/lighteval/tasks/tasks/dyck_language.py` scores with inspect_ai's `exact()`:
```python
from inspect_ai.scorer import exact
...
scorer=exact(),
```
In every inspect-ai release from 0.3.27 (Sep 2024) through 0.3.252, `exact()` compared sets of words: order and token multiplicity were discarded before comparison (upstream UKGovernmentBEIS/inspect_ai#4618, fixed by #4619, first released in 0.3.253). Dyck targets are space-separated sequences of closing brackets, and order is the property being tested, so under inspect-ai <= 0.3.252 a wrongly-ordered closing sequence with the right bracket types scores 1.0.
## To Reproduce
On inspect-ai 0.3.252:
```python
from inspect_ai.scorer._classification import max_exact_score
print(max_exact_score("} ] )", [") ] }"])) # wrong bracket order
print(max_exact_score("hello world", ["world hello"])) # word order
print(max_exact_score("hello world", ["hello world"])) # control: identical
print(max_exact_score("bananas", ["hello world"])) # control: unrelated
```
```
1.0
1.0
1.0
0.0
```
Same script on the 0.3.253 wheel: the first two print 0.0, controls unchanged.
## Expected behavior
A wrongly-ordered closing sequence scores 0. Requiring `inspect-ai >= 0.3.253` for these tasks (or otherwise ensuring the fixed scorer is used) would do it. Happy to open a PR for the pin bump. Results produced with older pins can only be inflated, never deflated, because the old comparison marks more answers correct.
## Version info
lighteval `main`; inspect-ai 0.3.252 vs 0.3.253; Python 3.13.5, Windows 11.
*(Claude helped write this up; I ran the repro myself, on both wheels.)*
Contributor guide
No contributing guide indexed for this repository
Research direction
Start at src/lighteval/tasks/tasks/dyck_language.py and inspect the project’s inspect-ai dependency declaration. Reproduce the scorer behavior with inspect-ai 0.3.252 and 0.3.253, then verify the dependency uses a version containing the upstream fix and that wrongly ordered Dyck sequences no longer score as correct.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- testing-qa
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 72/100