InternLM / InternLM/OVO-S-Bench
score.py: extract_answer() caps letters at A–E, but there are 6–7-option questions with answer F/G
- Dominant language
- Python
- Stars
- 55
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
### Bug
[`score.py::extract_answer()`](https://github.com/InternLM/OVO-S-Bench/blob/main/score.py#L44) hard-codes `[A-E]` in all of its regex patterns
(docstring: *"the bench uses at most 5 options"*). But the released data has questions with **up to 7 options (A–G)**. Any question whose correct answer is **F or G** can never be extracted, so it's scored wrong for every model regardless of output.
```python
from score import extract_answer
extract_answer("F") # -> '' (expected 'F')
extract_answer("Answer: F") # -> ''
```
Affected (release parquet, 1695 rows / 1722 query instances)
- Option counts: {2:454, 3:131, 4:1006, 5:1, 6:101, 7:2} → 103 rows have 6–7 options.
- 25 query instances have a correct answer of F or G, all in subcategories 1.3.1 and 2.1.2 (e.g. id=1.3.1_21, answer F = "Translating backward"; id=2.1.2_0 has both F and G). These bias the L1/L2 and Overall numbers low.
### Fix
Replace [A-E] with [A-H] (or at least [A-G]) in each pattern and update the docstring. The options are already rendered to the model as A–G, so only the scorer's extraction needs to change.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.