Multi-capture / multi-view fusion for per-ramp recall: correspondence, agreement-as-arbiter, 3D (RampNet 2.0)

Motivation (RampNet 2.0, blue-sky)

Mapillary and GSV capture the same physical curb ramp / intersection many times — multiple passes, angles, lanes, dates, and years. Today the pipeline throws this away: the auto-labeler thins to ~30 m spacing (min_spacing_m: 30, see benchmark/clovis/sample.json) precisely to avoid re-running the model on redundant captures. This issue tracks the opposite bet — treat the redundancy as independent evidence and fuse it — as a path toward the RampNet-2.0 goal of pushing per-ramp precision and recall toward the imagery's ceiling at once.

This moves the question from "which model is best on a single image?" (the benchmark frame) to "what's the best per-ramp decision the system can make given every capture of that ramp?" — which is the deployment / north-star frame, and is the natural home for the per-ramp recall unit #38 already wants.

Why redundancy is an advantage

• Recall by disjunction (the big win). A ramp occluded by a car/pedestrian, glare, or a bad angle in one pass is often clean in another. Detect per-capture, union at the physical-ramp level → recall rises monotonically. Independent observations, not synthetic augmentation. Aligned with recall-first (FN = permanent loss).
• Precision by agreement (a free geometric arbiter). A false positive — driveway, shadow, concrete patch — rarely reproduces across many angles/dates. Requiring k-of-n agreement across captures is an arbiter with no extra model, and k is a tunable P/R knob. Complementary to the crop-classifier arbiter in #35: geometry filters what the classifier would otherwise handle.
• Map coordinates, not pixels. Triangulating a ramp's ground position gives Project Sidewalk what it actually wants to place on a map, and doubles as the correspondence key (detections that converge on the same ground point are the same ramp).
• Temporal signal. Captures span years (clovis is 2019). Persistence across dates boosts confidence; appearance/disappearance flags construction or new builds (cf. the temporal-consistency filter from closed #11).

The correspondence problem — and modern tools for it

Matching detections across captures to the same physical ramp is the core difficulty. Three tiers, cheapest first:

Tier 1 — bearings-only triangulation (MVP, no reconstruction). In a 360 equirect pano the detection's pixel column directly gives its azimuth relative to camera_heading (which the records carry). Each detection → a world-space bearing ray from the camera lat/lng; rays from multiple captures intersect at the ramp. Needs only heading + pixel-azimuth — notably it does not need camera_pitch/camera_roll (both null in the Mapillary data). This is dedup_points (#43) lifted from within-pano to world coordinates.

Tier 2 — feed-forward multi-view geometry (the modern correspondence engine). The 2025 step-change is pose-free feed-forward reconstruction: VGGT (Visual Geometry Grounded Transformer, CVPR 2025 best paper) takes 1→hundreds of images and predicts camera poses + dense depth + point maps + 3D point tracks in one pass, replacing classical SfM+MVS; MASt3R/DUSt3R and Splatt3r are the same lineage. Because they estimate geometry from pixels, they sidestep Mapillary's variable crowd-SfM pose quality (the thing classical COLMAP chokes on). Feed a per-intersection bundle → every capture posed in a shared frame + per-pixel 3D → lift 2D detections to 3D → correspondence = clustering in 3D.

Tier 3 — semantic 3D / Gaussian Splatting (blue-sky, detect-and-measure). Note up front: 3DGS consumes correspondence+pose, it doesn't solve it — it's a rendering/scene representation on top of the geometry from Tier 2. Where it earns its place:

• Detect the ramp as a 3D entity, dissolving the N-views-reconcile problem — semantic/language-embedded splatting is mature (OpenSplat3D, CVPR'25 open-vocab 3D instance seg; LangSplatV2; GS4City with city-model priors).
• Occlusion-free canonical renders to feed the detector — render a clean orthographic top-down corner at a chosen scale (attacks both the ~20 px tiny-object problem and occlusion; #46/#47).
• Metric geometry → measure slope / width / tactile pad, i.e. the leap from "find + tag" to "rate ≥ humans." No single-image detector can do this.

Honest costs / open questions

• It fights the current cost strategy. Un-thinning is where the recall comes from, but it multiplies inference cost by local capture density (5–50× on popular roads). Same axis as #38 (sampling density), with a fusion step.
• Benefit is uneven. Dense urban cores have many captures; rural/residential were driven once. Strong where coverage is dense, nothing where it's sparse — measure benefit as a function of local capture count.
• Per-scene 3DGS doesn't scale (minutes–hour/intersection; clovis alone had 72,776 source records). Only feed-forward/generalizable variants are tractable, and their quality on 2–5 sparse, wide-baseline, multi-date street captures of one corner is unproven for this task — the sparse-view street survey flags off-driven-lane synthesis as still poor, which is exactly where a far-corner ramp lives.
• 3D is a multiplier, not a replacement. Dominant RampNet errors are "didn't fire" / "fired on concrete," not "mislocalized." Union recall is still bounded by "detected in ≥1 view" — the per-view detector still matters.
• It changes the evaluation unit to per-physical-ramp, keyed by world coordinate — a different (better, more deployment-honest) GT model, and exactly #38's unit.

Reframes the model comparison too

Cross-capture agreement would kill much of the chat-VLMs' false-positive flood (won't reproduce across angles) and recover some misses — so multi-view fusion could narrow the RampNet-vs-VLM gap, since the VLMs' main failure mode is precisely what agreement suppresses. "Best single-image model" and "best model in a multi-view system" may not be the same ranking.

Where it sits

Spans both repos (like the city-split boundary): enumeration / fetching / reconstruction / fusion mechanics are sidewalk-auto-labeler's job ("what are the ramps here"); the 3D detection + per-ramp evaluation (does fusion actually raise recall, at what cost) is RampNet's. Complements #35 (agreement = free arbiter), builds on #38 (per-ramp recall / density), generalizes #43 (within-pano dedup), and relates to #25 (resolution). Filing as a research direction / epic; not committing to build — the cheap Tier-1 experiment on the existing benchmark is the natural first probe.

Suggested first steps

1. Tier-1 probe on richmond/clovis (Mapillary, have heading): are there enough overlapping captures per ramp for triangulation to help? Quantify overlap distribution.
2. Measure oracle per-ramp recall under capture-union vs single-capture (reuses #38's per-ramp framing).
3. If overlap is thin, this is a density story (#38); if rich, prototype agreement-as-arbiter and compare FP-per-find against the #35 cascade.

Refs: #38, #35, #43, #25, #46, #47.

Sources: VGGT (CVPR'25 best paper) · sparse-view street GS survey · feed-forward driving GS · cross-view geo-localization via 3DGS · LangSplatV2 · GS4City

Start with benchmark/clovis/sample.json and the existing dedup_points work referenced in #43. Run the Tier-1 probe on richmond/clovis using the available capture headings, then measure overlap and oracle per-ramp recall for capture unions versus single captures. Done means reporting whether overlap is sufficient to justify agreement-based fusion and how recall changes with capture count.