open-compass / open-compass/opencompass

`gsm8k_postprocess` breaks numeric answers at thousands separators (`\boxed{$9{,}500}` → `500`, `\boxed{8,000}` → `000`)

Open Beginner friendly
#2,647 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7.5k
Forks
869
Avg merge
17h 52m
Merged PRs (30d)
13

Description

Minimal reproduction:

from opencompass.datasets.gsm8k import gsm8k_postprocess

measured (opencompass installed from PyPI, 0.5.4):

input output
\boxed{9500} 9500
\boxed{$9{,}500} 500
\boxed{8,000} 000
\boxed{10,000} 000
The answer is $9{,}500. 500

Impact: any answer containing a comma-separated number is extracted as a different number, and the
failure is silent — it is indistinguishable from the model answering wrongly. In our GSM8K run
(300 items, deepseek-flash, one leg per harness) this was one of the measurable sources of OpenCompass's
losses against the other two harnesses on the same raw answers: the same responses were graded correctly
by the other extractors.

Also worth noting: the same postprocessor handles \boxed{...} fine when the number has no separator,
so the defect is specifically the , inside a numeric token.

Suggested direction: treat , as a thousands separator when it sits between digits
(i.e. strip separators from the numeric token instead of splitting on them), or reuse the numeric
normalisation the metric applies.

Note on LaTeX: models often emit \,{,}-style escapes (9{,}500) precisely to avoid a math-mode
space, so this input shape is common, not exotic.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Inspect opencompass.datasets.gsm8k.gsm8k_postprocess and its numeric extraction path. Reproduce the listed boxed and prose inputs, then verify that comma-separated values retain their full numeric value while unseparated answers remain unchanged. Done means all shown numeric forms are extracted correctly without regressing the existing 9500 case.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
84/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.