huggingface / huggingface/lighteval

[BUG] dyck_language scores with inspect_ai's exact(), which ignored word order in every release up to 0.3.252

Open Beginner friendly
#1,329 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Python
Stars
2.5k
Forks
555
Avg merge
1d 6h
Merged PRs (30d)
1

Description

## Describe the bug

`src/lighteval/tasks/tasks/dyck_language.py` scores with inspect_ai's `exact()`:

```python
from inspect_ai.scorer import exact
...
scorer=exact(),
```

In every inspect-ai release from 0.3.27 (Sep 2024) through 0.3.252, `exact()` compared sets of words: order and token multiplicity were discarded before comparison (upstream UKGovernmentBEIS/inspect_ai#4618, fixed by #4619, first released in 0.3.253). Dyck targets are space-separated sequences of closing brackets, and order is the property being tested, so under inspect-ai <= 0.3.252 a wrongly-ordered closing sequence with the right bracket types scores 1.0.

## To Reproduce

On inspect-ai 0.3.252:

```python
from inspect_ai.scorer._classification import max_exact_score

print(max_exact_score("} ] )", [") ] }"])) # wrong bracket order
print(max_exact_score("hello world", ["world hello"])) # word order
print(max_exact_score("hello world", ["hello world"])) # control: identical
print(max_exact_score("bananas", ["hello world"])) # control: unrelated
```

```
1.0
1.0
1.0
0.0
```

Same script on the 0.3.253 wheel: the first two print 0.0, controls unchanged.

## Expected behavior

A wrongly-ordered closing sequence scores 0. Requiring `inspect-ai >= 0.3.253` for these tasks (or otherwise ensuring the fixed scorer is used) would do it. Happy to open a PR for the pin bump. Results produced with older pins can only be inflated, never deflated, because the old comparison marks more answers correct.

## Version info

lighteval `main`; inspect-ai 0.3.252 vs 0.3.253; Python 3.13.5, Windows 11.

*(Claude helped write this up; I ran the repro myself, on both wheels.)*

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at src/lighteval/tasks/tasks/dyck_language.py and inspect the project’s inspect-ai dependency declaration. Reproduce the scorer behavior with inspect-ai 0.3.252 and 0.3.253, then verify the dependency uses a version containing the upstream fix and that wrongly ordered Dyck sequences no longer score as correct.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
testing-qa
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.