Set up a harness for experimenting with newer detection models/backbones

Motivation

RampNet's paper state is preserved at v1.0-iccv2025, and the deployment side is now exercising the model on imagery beyond GSV — notably Mapillary 360 panos (Richmond, VA), where image quality is generally lower (consumer rigs, motion blur, re-compression). Newer backbones and foundation models have landed since the paper was trained, and it would be good to know systematically whether any of them improve curb-ramp detection — especially on lower-quality imagery.

Proposal

Set up a repeatable experiment path so trying a newer model is a config change plus a training/eval run, not a fork:

• Benchmark of record: the corrected gold-set evaluation (#9) as the fixed comparison target; report P/R at the operating threshold plus AP, alongside the committed v1.0-iccv2025 numbers.
• Model swapping: make the backbone/architecture configurable in stage_two/train.py (builds on warm-start support, #5), keeping the keypoint-head + heatmap contract stable so the evaluators and export path keep working.
• Candidates to seed the queue: newer ViT/DINO-family backbones with the existing head, and (separately scoped) a look at whether open-vocabulary or VLM-based detectors are competitive zero/few-shot.
• Results tracking: one results table (repo doc or issue thread) with env spec per run (#6), so experiments accumulate instead of scattering.
• Cross-domain eval: include a lower-quality-imagery slice (e.g. a validated Mapillary sample from the Richmond deployment) so improvements are measured where deployments actually hurt.

Happy to discuss scope — filing so the idea has a home while the Mapillary deployment work is fresh.

Follow-ups from the model-comparison harness (PR #34)

The VLM-comparison harness (RampNet vs Gemini/Qwen) landed via PR #34; these are the deferred increments it leaves for this issue:

• Wire the live Qwen3-VL call. Done in #36. QwenDetector loads the checkpoint once in _ensure_ready (AutoModelForImageTextToText, device_map="auto" with a single-device fallback) and runs the same tiled path as Gemini. qwen-vl-utils turned out to be unnecessary — Qwen3-VL's chat template takes PIL images directly. Default id is Qwen/Qwen3-VL-8B-Instruct (Qwen/Qwen3-VL was never a real repo id). Launcher scripts/model_comparison/run_qwen.slurm + a Hyak runbook in docs/model_comparison.md. The 8B/32B benchmark runs themselves are still to do — tracked separately below.

• Run the Qwen benchmark on Hyak. 8B first (sbatch scripts/model_comparison/run_qwen.slurm), then 32B (QWEN_MODEL=Qwen/Qwen3-VL-32B-Instruct sbatch --gpus=2 ...), over richmond + bend. Open question the run answers: Gemini Flash/Pro were FP-heavy (richmond F1 0.63–0.66 vs RampNet 0.855) — does a larger open model get the precision they lacked? Results come back through .model_cache/, whose key contains nothing machine-specific, so the final table is produced locally with no GPU.

• Qwen box normalization — resolved, but not as described here: this item was based on a wrong premise. Qwen3-VL emits bbox_2d normalized 0–1000, not absolute pixels. Absolute-pixels-of-the-resized-image was Qwen2.5-VL's convention. A 0–1000 box is resolution-independent, so the processor's 28-multiple smart-resize cannot shift it and there is no processed-size to thread through — the drift this item worried about cannot occur. Verified empirically rather than from the docs, by rendering one view at three sizes:

  view render	boxes returned	max coordinate
  512×512	1	345
  1024×1024	1	365
  1400×1400	1	360

  Absolute-pixel output would have scaled with the image (~1400 at the largest). It didn't. qwen_boxes_to_points now takes an explicit coord_space (norm1000 | pixels) inferred from the model id, with --qwen-coord-space to override, so a Qwen2.5-VL checkpoint still works. It is deliberately not auto-detected: at a 1024px view the two conventions differ by only 2.4%, so a wrong guess doesn't crash — it introduces a small systematic localization bias a P/R table won't reveal. (#36)

• Calibrate the reprojection rig against the first live VLM run. Tune fov_h_deg / n_yaw / pitch_deg so ramps land near-centered in some view, minimizing seam-truncation false positives; report --tiling perspective vs --tiling none side by side. The new dump_detections.py overlays quantify the nadir/hood waste: at pitch_deg=-30 the bottom ~40% of every view is the capture vehicle's hood and the black nadir cap, so roughly a third of every paid call is spent on pixels that cannot contain a curb ramp.

• Benchmark open-weight detectors and pointing models — split out to #39. Every model tested so far is a chat VLM localizing as a side skill, and all are FP-heavy (119–293 FPs vs RampNet's 9). #39 covers OWLv2 / Grounding DINO (purpose-built open-vocabulary detectors, and the only candidates that carry real confidences → AP and PR curves) and AI2's Molmo (pointing-native, so no box→center reduction).

• Add the clovis split once the auto-labeler hands back its bundle; the harness is city-generic (needs records.jsonl + verdicts.json + panos/).

Low-priority note: legacy verdicts.json without a no_missed field are now rejected by the harness's pre-flight validate_bundle with a clear message — a 1-line .get() in build_ground_truth would restore collect()'s "legacy = trusted" behavior if the harness is ever pointed at an old-schema bundle.

Start with the unchecked Qwen benchmark path in scripts/model_comparison/run_qwen.slurm and the Hyak instructions in docs/model_comparison.md; run the 8B model over Richmond and Bend before the 32B variant. Review the experiment configuration in stage_two/train.py and the existing harness from PRs #34 and #36. Done means reproducible runs and results recorded against the corrected gold set and v1.0-iccv2025 numbers.