Experiment: hybrid RampNet + open VLM (Qwen3-VL) — ensemble / cascade / distillation

Motivation

RampNet is strong but not perfect — on the deployment benchmark it sits around P ~0.95 / R ~0.80, so ~1 in 5 real curb ramps is still missed. General VLMs now do bounding-box detection, and the model-comparison harness (#20, PR #34) lets us score them on the same ground truth. The question this issue tracks: can a hybrid of RampNet + an open VLM beat RampNet alone — at a cost an open-source, academically-hosted project can actually afford?

Project Sidewalk is open source, runs on academic servers, and has very little money, so "just call a paid VLM per pano" is off the table at deployment scale. That constraint isn't a blocker — it shapes the design.

The cost reframe: self-host the VLM

• Gemini (paid API) → useful as a reference baseline, but never a deployment component (per-call cost × millions of panos).
• Qwen3-VL (open weights) → self-hostable on makelab2 (A40, 48 GB, 24 TB) or Hyak; inference cost is GPU-hours, not dollars. This makes Qwen — not Gemini — the real hybrid partner.
• Tiling wrinkle worth testing: the harness feeds the VLM 1024×1024 perspective tiles, not the full 4096+ pano. The "A40 OOMs Qwen at native res" concern (why Qwen was earmarked for Hyak) may not apply to tiled inference — so makelab2 could be self-sufficient here. Verify empirically.

Approaches (with the standard names)

Decision gate — do this first (cheap, no training)

A hybrid only helps if the two models make different errors. Before building anything, run the complementarity analysis on the existing model-comparison run (cached detections + derived GT):

• Oracle-union recall — recall if a ramp counts as found when either model finds it. Compare to RampNet's solo recall.
• The "RampNet-miss ∩ VLM-hit" set — real ramps RampNet misses that the VLM catches (the upside), and the FP cost of taking the VLM's extra detections (the downside).
• Verdict: if union recall materially exceeds RampNet's ~0.80 without a FP blowup → prototype a cascade. If not → it's a distillation story (VLM cleans the dataset), or not worth it.

Early data point (in-progress run): Gemini-3.6-flash tiling ≈ P 0.76 / R 0.65 on 70 richmond panos — below RampNet — so a naive union would add FPs. Whether the errors are complementary is the open question. Qwen numbers TBD (needs wiring, see #20).

Compute

• makelab2 — A40 (48 GB), 24 TB; self-host Qwen3-VL (test tiled inference here first). Needs env setup (py3.12 + CUDA torch + transformers pin).
• Hyak — for heavier runs / native-res if tiling proves insufficient.
• Gemini — paid reference baseline only, via the harness.

Relation to other work

Builds directly on #20 (model-comparison harness) and PR #34 (docs/model_comparison.md, scripts/model_comparison/). The complementarity analysis reuses that harness's cached detections and rampnet/detection_eval.py ground truth — no new infra. Wiring Qwen3-VL (the prerequisite for a Qwen hybrid) is already a checklist item on #20.

Next steps

1. Complementarity analysis on the current Gemini run (defines whether to proceed).
2. Wire + self-host Qwen3-VL on makelab2 (test whether tiled inference fits the A40); rerun complementarity with Qwen.
3. If promising: prototype the cheapest viable hybrid (likely a confidence-gated cascade or a distillation pass), measure vs RampNet-alone on the benchmark.

Filing to give the idea a home; not committing to build until the complementarity gate says it's worth it.

Start with the cached detections and derived ground truth from scripts/model_comparison/ and rampnet/detection_eval.py, using the existing Gemini run. Measure oracle-union recall, RampNet-miss/VLM-hit cases, and false-positive cost, then report whether the results justify a cascade, distillation pass, or stopping; Qwen3-VL wiring and tiled A40 inference are later steps tied to #20.