Score the frontier releases: Claude Fable 5/5.1 and GPT-6 Astra (#122 roster refresh)

Why

"Did you try the newest models?" is the first question a reviewer asks about a benchmark whose
headline is "RampNet beats every general-purpose model." Two frontier releases landed in the last
week, so the answer is currently "no", and the registry in #122 exists precisely so that answer is
cheap to refresh rather than a bespoke project each time.

This issue covers two candidates that are not the same amount of work. They are independent —
land either without the other.

What we already know, so nobody expects a surprise

Three separate results in this benchmark say general capability does not transfer to this task:

• Qwen3-VL 32B scores worse than Qwen3-VL 8B — 0.018 F1 against 0.161 on laurens_gsv.
  Scaling flipped the failure mode instead of fixing it: the bigger model stops firing.
• claude-opus-5 at effort high loses 0.068 F1 to the same model at effort low on
  annapolis (0.520 vs 0.588), burning 127,227 thinking tokens to do it. Effort is an
  operating-point dial, not a quality lever (#123).
• Open-detector recall is mostly density. OWLv2 and Grounding DINO post 0.85–0.88 recall on
  laurens_gsv at 0.028 precision — 6,000+ false positives. Recall without precision is coverage,
  not detection.

Finding small, distant curb ramps in a 4096x2048 equirectangular panorama is a localization
problem. Frontier reasoning gains have not touched it so far. The prior is that a new frontier
model lands near gemini-3.1-pro (0.28–0.57 F1 depending on split), not near RampNet (0.66–0.91).

That prior is the reason to run it, not a reason to skip it. A negative result here is
publishable content (see CLAUDE.md), and it is the cheapest possible answer to the reviewer
question. What we must not do is spend $150 discovering it.

Protocol (preregistered, so the result is not a fishing expedition)

1. One split first: annapolis. It is the established beachhead for paid legs — four Claude
   legs already published there, with tests/test_claude_annapolis_leg.py recomputing every
   published number from committed detections with no API key, no GPU and no .model_cache.
   A new leg inherits that guarantee.
2. Expand to the full 10 splits only if the model clears gemini-3.1-pro-preview (0.567 F1)
   on annapolis. Below that it is a data point, not a challenger, and one split says it.
3. standing=False in the roster until it has been run on enough splits to join the tables —
   the same rule the Claude and gemini-3.7-flash legs follow.
4. Record spend at run time via --usage-log (#143). Token counts are write-once: a re-run reads
   the detection cache, makes zero calls, and can never reproduce them.

Item A — Claude Fable 5 / Fable 5.1 (small)

Same detector, same credentials, same rig. ClaudeDetector already takes an arbitrary model id.

Both exist on our Vertex path and are one project setting away. Probed 2026-09-04 against the
global endpoint with the same ADC the Gemini and Claude legs use:

claude-fable-5     403  "Access to this model requires data sharing to be enabled for
                         publisher 'anthropic'. Please set
                         PublisherModelConfig.data_sharing_enabled_provider to 'anthropic'
                         via the setPublisherModelConfig API to use this model."
claude-fable-5-1   403  (same message)
claude-fable-5.1   404  "Publisher model ... was not found"   <- wrong id spelling, not a real model
claude-opus-5      200  OK

The 404 on the dot-spelling is what makes the 403s informative: the gate does not fire blindly
on the anthropic publisher, so a 403 means Vertex resolved a real model id behind it. Both Fable
ids are real and reachable once the project enables data sharing for that publisher.

Tasks:

• Decide whether to enable data_sharing_enabled_provider = anthropic on the project. This is
  a project-level configuration change and belongs to @jonfroehlich, not to a script.
• Add pricing.py rows. Fable 5 is $10/$50 per MTok on the first-party card (2x claude-opus-5);
  read the Vertex Anthropic rate card rather than assuming parity, and stamp as_of.
  Fable 5.1's price is unknown to us — it must be read, not inferred from Fable 5.
• Roster rows, standing=False, published_as per leg if effort is pinned.
• Run annapolis, effort low. low matters more here than on Opus: Fable's thinking is
  always on (thinking: {type: "disabled"} is a 400 and budget_tokens was removed), so the
  near-zero-thinking floor that makes our Opus leg cheap does not exist. Expect the cost band
  to be wider than a flat 2x.
• Extend tests/test_claude_annapolis_leg.py to cover the new leg.

Estimated cost, annapolis only (125 panos x 6 views = 750 calls): our measured claude-opus-5
effort-low leg was $8.94. At 2x token pricing that is ~$18 floor, and plausibly ~$25
if always-on thinking runs longer than Opus at low. Both arms of Laurens would add ~$26–36.

Item B — OpenAI GPT-6 Astra (new provider, real work)

Announced 2026-09-03; API id gpt-6-astra; 1M-token context; priced at 2.5x GPT-5.6 Sol. Limited
preview for trusted partners at announcement, general availability 2026-09-05.

There is no OpenAI path in this repo at all. This is not a roster row, it is a provider:

• OpenAIDetector in scripts/model_comparison/detectors.py, structured-output box tool
  mirroring CLAUDE_BOX_TOOL, sharing the same DETECTION_PROMPT so the leg stays comparable.
• A credential story. Every paid leg so far rides Vertex + ADC; this one needs a separate key
  and a decision about where it lives.
• pricing.py rows (read, with as_of) and an openai entry in PAID_PROVIDERS.
• Verify the coordinate space before trusting any number. dump_detections.py on a handful
  of panos, exactly as was done for Claude (#122) before its numbers were read. Qwen needed
  --qwen-coord-space and Molmo needed --molmo-coord-scale; assuming a convention is how a
  leg silently scores a systematic offset instead of a detector.
• Confirm API access is actually available on our account tier — Astra's rollout is staged
  through OpenAI's cybersecurity program first, so availability on 09-05 is announced, not
  verified by us.
• Run annapolis, then apply the expansion rule above.

Cost: unknown. "2.5x GPT-5.6 Sol" is not a number we hold, and this repo's rule is that a price
goes in the table with the date it was checked and the page it came from, never from memory.
Price it before running it.

Out of scope

Re-running the nine already-scored splits for models that do not clear the annapolis bar. The
expansion rule above exists to prevent exactly that.

Provenance

Vertex probes run 2026-09-04 from the repo's own credential path. Benchmark numbers quoted above
are from docs/model_comparison.md and are re-derivable from committed detections.

🤖 Generated with Claude Code (claude-opus-5[1m])

Start with scripts/model_comparison/detectors.py, pricing.py, the roster in #122, and tests/test_claude_annapolis_leg.py. First review the existing Claude leg and run dump_detections.py on sample panoramas to verify coordinate space. Done means the required pricing, provider and roster entries exist, credentials and access are confirmed, annapolis runs with usage logging, and the relevant tests recompute the committed detections.