Prompt study for the challenger VLMs: image exemplars, elicited confidence, query ensemble

Motivation

The prompt is the single biggest uncontrolled variable in the model comparison, and right now it's fixed at one untuned setting. Every prompted challenger gets the same zero-shot, text-only prompt (DETECTION_PROMPT in scripts/model_comparison/detectors.py): one definition sentence + a short negative list, temperature 0, single pass, no in-context examples. That's a reasonable prompt; we have no evidence it's a good one, and VLM grounding/counting is notoriously prompt-sensitive.

So the current results measure "these VLMs, with this one prompt" — not the VLMs' capability. Before the paper concludes general models underperform, we should establish how much of the gap is the prompt.

Experiments

1. Few-shot image exemplars. Add 2–4 labeled crops to the prompt: real curb ramps and the exact confusers the models keep boxing (driveway apron, painted crosswalk, stairs). A curb ramp is a functional, context-defined class that's hard to pin in words; visual in-context examples are the highest-expected-value lever. Applies to the chat VLMs (Gemini/Qwen), and to Molmo.
2. Elicit per-box confidence. Ask the chat VLMs to emit a 0–1 confidence per box. Today they're pinned at one operating point with no PR curve, while OWLv2/GDINO get a tunable sweep — a real structural disadvantage in the comparison. A score gives chat VLMs a PR curve + threshold sweep too, and lets the recall-first framing apply to them.
3. Query ensemble for the open detectors (cheap, minutes): "curb cut", "wheelchair ramp at a crosswalk", a multi-query ensemble. --owlv2-query / --gdino-query already exist. Tells us whether their ceiling is the query or the model class (docs/model_comparison.md Next increments #4).
4. (stretch) Two-stage / CoT and self-consistency — find crossings then ramps; vote over a few samples at T>0 vs one greedy pass.

Fairness constraint (important)

Tuning the prompt on richmond/bend is tuning-on-test — the exact sin docs/model_comparison.md already flags for the threshold sweep. To keep the comparison honest against RampNet's fixed operating point: tune the prompt on a dev split (clovis, once the third-city companion issue lands its GT) and report richmond/bend as a held-out envelope, or pre-register a small prompt set and report best-of-N as an explicit optimistic upper bound.

Cost / mechanics

The prompt is part of the detector signature (_VLMDetector.signature), so each variant invalidates the detection cache and re-bills. Scope it: sweep prompts on one city or a --limit subset first, pick the winner, then run full. Gemini is paid; Qwen/Molmo are cluster GPU-hours (Hyak).

Why it's worth it either way

• A best-of-N few-shot prompt still trails RampNet by ~0.2 F1 → the paper's claim gets much stronger ("we tried hard to make general models work and they still don't").
• It closes the gap → the story changes, and we want to know before a reviewer does.

Refs #39, #20, #35.

Start in scripts/model_comparison/detectors.py at DETECTION_PROMPT and review the comparison guidance in docs/model_comparison.md, including the existing query flags and threshold-sweep constraints. Define a scoped prompt experiment before implementing the selected variant. Done means results are reported for the dev split and held-out richmond/bend data, with costs and any best-of-N optimism made explicit.