Released 5-choice MCQ prompt drops the options — with a confirming experiment

Thank you for releasing Spatial-IQ with full code, prompts, and per-item predictions — that openness is what made the checks below possible, and we hope they help with the revision your code comments mention.

1. The released 5-choice prompt never shows models the options

In inference/prompt_mcq_Type1.json, task_prompts.main contains no {choices} placeholder and no OPTIONS: block (unlike the 3- and 4-choice files, which define and use CRITICAL_INSTRUCTION_MAIN — the 5-choice file appears to have missed that update). The call site does main_task_prompt.format(choices=choices_str), and Python's str.format silently discards unused keyword arguments — so the five integer options are computed and then dropped. The same template also announces "exactly 6 images" while the code sends one. Rendering the exact final prompt for a sample item confirms the option values appear nowhere in it.

This is visible in the released predictions:

• All eight 5-choice model results sit within 0.7 points of exactly 20% (less spread than sampling noise), while the same models score 29–49% at 3 and 4 choices.
• No model's 5-choice accuracy depends on where the correct answer ranks among the five values (e.g. GPT: χ² = 4.2, p = 0.39), while the same model shows enormous rank dependence where options are visible (3-choice: χ² = 727.6, p ≈ 4e-156).
• Many 5-choice GPT replies state the problem directly: "Please provide images 2–6 (options A–E)."

Confirming experiment. We reran 200 rank-stratified task_main items with your model string and settings, changing only the prompt (the 4-choice OPTIONS: block added). Accuracy moves off chance (27.0%, exact binomial p = 0.0167) and the rank-dependence signature appears (χ² = 46.3 on 4 df, p = 2.2e-09; permutation p < 5e-06), matching the shape of your own 4-choice results (65% when the correct answer is the smallest value, 5% when it is the largest). The unmodified released prompt, rerun the same day on the same items, stays flat (χ² = 2.0, p = 0.74).

Practical implication: the paper's 5-choice MCQ numbers (Table 2's chance-adjusted values and Figure 5's tests against a 20% null) reflect models answering without seeing the options.

2. Refusals are silently scored as answers

evaluation_mcq.py::_parse_letter falls back to the last standalone A–E token in a reply. On the refusal sentence above, that token is the "E" in "A–E" — in our rerun of the released prompt, 43 of 47 refusals were scored as the answer "E" with no flag. A refusal-detection guard before the fallback would prevent this.

3. Two statistical notes for the revision

• Distractor symmetry makes the correct value the median more often than 1/5. Because distractor offsets pool near-symmetrically around the correct value, a text-only "pick the median" rule scores 27.9% on the released 3,000 items (p ≈ 1e-25 vs a 20% null). Once options reach the models this shifts the effective chance baseline and interacts with per-model rank biases (the released 4-choice GPT results fall monotonically from 61.8% to 8.1% across ranks). Sampling distractors asymmetrically, or reporting rank-stratified accuracy, would remove the artifact.
• The consistency ("summation") check has an image-blind ceiling above the human baseline (82.1% achievable without the image vs 69.1% human), and 3 of the 7 adjacent-pair significance claims do not survive false-discovery correction on the released per-item data (p up to 0.58).

We work on evaluator validity and are happy to share the full reproduction package — scripts, rendered prompts, raw API responses, and a pre-registered analysis plan written before the confirming run. No response expected; we thought you would want these before the revision finalizes.

Read inference/prompt_mcq_Type1.json and evaluation_mcq.py::_parse_letter, then render a 5-choice prompt and inspect the released evaluation path. Done means the prompt visibly includes the choices, the image-count wording is reconciled with the call, and refusal outputs are not silently parsed as answers; separately verify the statistical concerns against the released per-item data.