OpenVLM: one-decimal Overall makes the board un-checkable, and hides that 8 boards' #1 leads are under 1.2 sigma

Publishing Overall to one decimal makes the OpenVLM leaderboard un-checkable, and hides that several boards' #1 leads are inside noise

Thanks for keeping OpenVLM.json public and complete — 285 models across 28 benchmarks is a genuinely useful artifact, and everything below comes from it alone.

This is a sibling of open-compass/opencompass#2574, which I filed today about CompassAcademic. The contrast between the two is the actual point of this issue, so I'll start there.

One decimal is the difference between checkable and not

On CompassAcademic, scores are published to two decimals. That was enough to recover every denominator without being told them: searching for all n consistent with round(100k/n, 2) across all published scores returned exactly 541 / 198 / 500 / 30 / 400 / 164 / 148 — the true dataset sizes. From there, standard errors and rank separation follow.

On OpenVLM, Overall is published to one decimal. I ran the same search over n in 1..4000 for every benchmark with ≥50 models:

benchmark	models	consistent denominators found
MMBench_TEST_EN_V11	284	3001
MMMU_VAL	284	3004
MathVista	284	3005
AI2D	284	3004
MMStar	284	3004
…	…	~3000 each

Roughly three thousand candidate denominators survive for every board — the constraint stops binding above n≈1000. No denominator can be recovered, so no reader outside the project can compute an error bar on any OpenVLM number. (MME and LLaVABench were excluded up front: they aren't percentages.)

Going to two decimals would cost nothing and would make the whole board independently checkable, exactly as CompassAcademic already is.

What can still be shown, without any denominator

The argument below needs no n at all — only an upper bound on it. Standard error decreases with sample size, so evaluating each board at the largest test set that appears anywhere on this leaderboard (SEEDBench_IMG, ~14,233) gives the smallest possible SE and therefore the most generous possible significance for the leader. If a lead is small there, it is smaller still at any realistic size.

benchmark	#1	lead over #2	SE at n=14,233	lead in σ (upper bound)
ChartQA_TEST	88.2	0.1	0.270	0.37
MMBench_TEST_EN_V11	88.5	0.1	0.267	0.37
MMBench_TEST_EN	88.9	0.1	0.263	0.38
POPE	90.5	0.1	0.246	0.41
QBench	80.5	0.2	0.332	0.60
MMBench_TEST_CN_V11	88.3	0.2	0.269	0.74
RealWorldQA	78.7	0.3	0.343	0.87
MMBench_TEST_CN	88.8	0.3	0.264	1.13

Eight boards have a #1 that is under 1.2σ ahead even under the most favourable denominator on the leaderboard, and four are under 0.4σ. At sizes typical of these benchmarks the same gaps are far smaller: at n=1500 a 0.1 pp lead is 0.12σ, at n=3000 it is 0.17σ.

Two boards also have a displayed tie at the top:

• MMBench_TEST_EN_V11: JT-VL-Chat-V3.0 and Gemini-2.5-Pro both 88.5
• POPE: InternVL2.5-26B-MPO and InternVL3-78B both 90.5

I want to be careful here, and this is exactly why the precision matters: at one decimal I cannot show these ties are exact. On CompassAcademic I could — one item was worth 0.505 pp there, so identical display forced an identical raw count. Here one item is worth roughly 0.03–0.11 pp, so a displayed tie may be hiding a real difference. The published table simply cannot order these models, which is a statement about the table rather than about the models.

Reproduce

import json, math, urllib.request, collections
d = json.loads(urllib.request.urlopen(
    "http://opencompass.openxlab.space/assets/OpenVLM.json").read())["results"]

cols = collections.defaultdict(list)
for model, rec in d.items():
    for bench, v in rec.items():
        if bench != "META" and isinstance(v, dict) and isinstance(v.get("Overall"), (int, float)):
            cols[bench].append((model, float(v["Overall"])))

N_MAX = 14233                      # largest test set anywhere on this leaderboard
for bench in ("MMBench_TEST_EN_V11", "ChartQA_TEST", "POPE", "MMBench_TEST_EN"):
    vs = sorted(cols[bench], key=lambda x: -x[1])
    top = vs[0][1]
    nxt = next(v for _, v in vs if v < top)
    se = math.sqrt((top/100) * (1 - top/100) / N_MAX) * 100
    print(f"{bench}: #1 {top} lead {top-nxt:.1f} pp -> at most {(top-nxt)/se:.2f} sigma")

Limits, stated

• Noise is assumed independent across models, and the published score is treated as truth. That is the standard framing; it is a model, not a fact.
• The σ figures are upper bounds, deliberately. I do not know the real denominators — that is the first half of this issue.
• Circular Eval on the MMBench series means a per-question outcome isn't a plain Bernoulli draw, so even with a known question count the binomial SE is an approximation there. Another reason the project's own numbers would be better than any outside reconstruction.

Suggestion

1. Publish Overall to two decimals (and ideally the denominator per benchmark). This is the small change, and it is what would let anyone verify the board the way CompassAcademic can be verified today.
2. Significance tiers or a CI column, once the first is in place.

Happy to open a PR for either, and equally happy to be told I've misread the scoring protocol — I'll close this myself and say so if that's the case.

Start with the published OpenVLM.json artifact and run the reproduction script in the issue to inspect the current Overall precision and benchmark records. Trace how this artifact is generated, then update the published Overall values to two decimals; done means the regenerated board preserves the results and allows the reported comparisons to be checked.