open-compass / open-compass/opencompass
`gsm8k_postprocess` breaks numeric answers at thousands separators (`\boxed{$9{,}500}` → `500`, `\boxed{8,000}` → `000`)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7.5k
- Forks
- 869
- Avg merge
- 17h 52m
- Merged PRs (30d)
- 13
Description
Minimal reproduction:
from opencompass.datasets.gsm8k import gsm8k_postprocess
measured (opencompass installed from PyPI, 0.5.4):
| input | output |
|---|---|
\boxed{9500} |
9500 ✅ |
\boxed{$9{,}500} |
500 ❌ |
\boxed{8,000} |
000 ❌ |
\boxed{10,000} |
000 ❌ |
The answer is $9{,}500. |
500 ❌ |
Impact: any answer containing a comma-separated number is extracted as a different number, and the
failure is silent — it is indistinguishable from the model answering wrongly. In our GSM8K run
(300 items, deepseek-flash, one leg per harness) this was one of the measurable sources of OpenCompass's
losses against the other two harnesses on the same raw answers: the same responses were graded correctly
by the other extractors.
Also worth noting: the same postprocessor handles \boxed{...} fine when the number has no separator,
so the defect is specifically the , inside a numeric token.
Suggested direction: treat , as a thousands separator when it sits between digits
(i.e. strip separators from the numeric token instead of splitting on them), or reuse the numeric
normalisation the metric applies.
Note on LaTeX: models often emit \,{,}-style escapes (9{,}500) precisely to avoid a math-mode
space, so this input shape is common, not exotic.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Inspect opencompass.datasets.gsm8k.gsm8k_postprocess and its numeric extraction path. Reproduce the listed boxed and prose inputs, then verify that comma-separated values retain their full numeric value while unseparated answers remain unchanged. Done means all shown numeric forms are extracted correctly without regressing the existing 9500 case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 84/100