Benchmark open-weight detectors and pointing models (OWLv2, Grounding DINO, AI2 Molmo) against RampNet

Motivation

The model-comparison harness (#34, #36) has now scored two proprietary VLMs and one open one against RampNet on the richmond split. Every general-purpose VLM has lost badly, and the newest entrant lost worst of all:

(richmond, 124 reviewed panos, perspective tiling, match radius 0.022. Bend and Qwen3-VL-32B are running.)

The pattern worth noticing: these are all chat models doing localization as a side skill. They are FP-heavy — 119–293 false positives against RampNet's 9 — which is what you get when a conversational model is asked to enumerate small objects. Before concluding "general models can't do this," we should test the model classes that are actually built for it. All of the following are open weights, i.e. free to run on Hyak.

Proposal — three candidates, in priority order

1. Open-vocabulary detectors: OWLv2 + Grounding DINO

• google/owlv2-large-patch14-ensemble
• IDEA-Research/grounding-dino-base

Text-prompted detectors, not chat models: give them "curb ramp", get boxes back. These are the standard strong open baseline for exactly this task, and they're small — minutes to run, no multi-GPU sharding.

They also unlock a capability the harness currently lacks. docs/model_comparison.md states "No AP for VLMs" because Gemini/Qwen boxes carry no calibrated score, so we only ever report a single operating point. OWLv2 and Grounding DINO emit real per-box confidences, which would give us PR curves, AP, and a threshold sweep for a non-RampNet model for the first time. That matters directly for the recall-first direction: a detector that can be tuned toward recall is worth far more than a chat VLM pinned at one operating point.

2. AI2 Molmo — points, not boxes

• allenai/MolmoPoint-8B — a dedicated pointing model
• allenai/Molmo2-8B (Qwen3-based) and allenai/Molmo2-O-7B (Olmo-based, fully open incl. data)

Molmo pioneered image pointing, and Molmo 2 shipped Dec 2025 with pixel-level grounding — its 8B reportedly beats the previous 72B.

Points are RampNet's native output format. Every model tested so far emits boxes that the harness reduces to centers, which is a documented caveat ("Box → point reduction" in docs/model_comparison.md). Molmo removes that reduction entirely, making it the most apples-to-apples comparison available. AI2 is also local/UW-adjacent, which is a nice angle if this ends up in a write-up.

3. Nice-to-have

Qwen/Qwen3-VL-32B-Instruct-FP8 if the bf16 32B run shows the gap is capacity-bound rather than structural.

Work required

The harness is already model-agnostic — tiling, scoring, caching, cluster launcher, and the Hyak runbook all carry over unchanged. Per model it's a small shim in scripts/model_comparison/detectors.py:

• OwlV2Detector / GroundingDinoDetector — transformers AutoProcessor + post_process_grounded_object_detection; boxes are absolute pixels with scores. Carry the confidence through instead of None (_VLMDetector._parse already supports a confidence slot, and score_pano orders greedy matching by it).
• Report AP / PR curves for these models via rampnet.metrics, and drop the "No AP for VLMs" caveat to "no AP for chat VLMs".
• MolmoDetector — parses <point x="..." y="..."> / <points> XML, coordinates normalized 0–100, not 0–1000. Verify the scale empirically the way Qwen's convention was verified (render one view at several sizes, confirm coordinates don't scale) and prove placement with dump_detections.py.
• Extend dump_detections.py to draw point-style predictions, not only boxes.
• Add each to --models and the results table; unit tests for the new parsers in tests/test_model_comparison.py.

Guardrail

Whatever is added must keep test_gemini_cache_key_is_frozen green — the detection cache holds thousands of already-paid Gemini detections, and a change to the shared signature silently re-bills them.

Refs #20, #36.

Start in scripts/model_comparison/detectors.py and review tests/test_model_comparison.py, docs/model_comparison.md, and dump_detections.py. Run the existing model-comparison tests first, then implement the listed detector parsers, confidence and point handling, model registration, reporting, visualization, and parser coverage while keeping test_gemini_cache_key_is_frozen green.