InternLM / InternLM/OVO-S-Bench

score.py: extract_answer() caps letters at A–E, but there are 6–7-option questions with answer F/G

Open
#1 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
55
Forks
0
PR merge metrics
No merged PRs in 30d

Description

### Bug

[`score.py::extract_answer()`](https://github.com/InternLM/OVO-S-Bench/blob/main/score.py#L44) hard-codes `[A-E]` in all of its regex patterns
(docstring: *"the bench uses at most 5 options"*). But the released data has questions with **up to 7 options (A–G)**. Any question whose correct answer is **F or G** can never be extracted, so it's scored wrong for every model regardless of output.

```python
from score import extract_answer
extract_answer("F") # -> '' (expected 'F')
extract_answer("Answer: F") # -> ''
```

Affected (release parquet, 1695 rows / 1722 query instances)

- Option counts: {2:454, 3:131, 4:1006, 5:1, 6:101, 7:2} → 103 rows have 6–7 options.
- 25 query instances have a correct answer of F or G, all in subcategories 1.3.1 and 2.1.2 (e.g. id=1.3.1_21, answer F = "Translating backward"; id=2.1.2_0 has both F and G). These bias the L1/L2 and Overall numbers low.

### Fix

Replace [A-E] with [A-H] (or at least [A-G]) in each pattern and update the docstring. The options are already rendered to the model as A–G, so only the scorer's extraction needs to change.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.