Make RampNet rig-native for open imagery: retrain on Mapillary/Panoramax hard positives mined from multi-view consensus, precision check first

Why now

Project Sidewalk's auto-labeling is expanding its investment in Mapillary and other open imagery — Bayonne just became the first Panoramax city (ProjectSidewalk/SidewalkWebpage#5200). RampNet has never trained on a non-GSV panorama: Stage 1's supervision (city inventories projected into GSV panos, and Project Sidewalk labels on GSV panos) is GSV-only.

The #86 thesis — detection is close to done, condition inference is the differentiator — was measured on GSV and on car-mounted Mapillary rigs, and it holds there. It does not hold on the rigs that dominate open imagery outside big-city car fleets. Read the scoreboard by rig rather than by city (docs/model_scoreboard.md, by-split F1, uniform scorer, operating point 0.55):

split	source	capture	F1
richmond	Mapillary	11000 px rig, mostly 2024	0.855
annapolis	Mapillary	8000 px	0.839
morgantown	Mapillary	4096 px	0.835
clovis	Mapillary	GoPro Fusion, 5760 px	0.801
bend / paterson / gainesville	GSV	Google rig	0.850 / 0.805 / 0.803
laurens_mapillary	Mapillary	GoPro Max, November	0.543
budapest_district5 †	Mapillary	consumer 360 rig (4% of GT above the horizon)	0.644

† held out.

#151 closed the loop on Laurens this week: the GSV arm over the identical footprint reads R 0.509 against Mapillary's 0.390, so rig and season are worth ~12 recall points, the deficit mostly survives anyway, and the conclusion was RampNet meeting an out-of-domain rig — with training data, not imagery, as the fix. Bayonne's 64k panos are GoPro Max, the Laurens rig. Under an open-imagery-first pipeline, the out-of-domain rig is the normal case.

So the order flips: the detector has to become rig-native before #86's measure/rate/tag work is worth doing on the imagery we are actually going to run.

Why the levers already on the board do not address this

• Operating point 0.30 (#54, #79, ProjectSidewalk/sidewalk-auto-labeler#20): +7.1 per-pano recall pooled, but the labeler's world-space eval (sidewalk-auto-labeler#27) puts the deployed gain at +0.4 to +3.2 points — multi-view fusion already cashes most of it. Flip the constant; it is not a rig fix.
• More of the same labels (#59): the epoch curve (#84) found no resolvable human peak after epoch 1 — the model saturates on Stage 1's GSV labels. Volume from the same domain will not move it.
• Higher input resolution (#25): forecast +10 points saturating ~0.875, measured on GSV plus one Mapillary rig, and entangled with rig. Worth running — as an arm of this retrain, not a separate campaign.
• Supervised transfer from Vistas (#126, #137): the cascade gate's ceiling was ~19 ramps.

The label source

A. Hard positives mined from multi-view consensus (#102). A fused site with ≥3 corroborating operational views is almost certainly a ramp (ghost check: other-pano support 0.967 for verdict-true vs 0.636 for verdict-false). Every nearby pano that produced no detection for it is a miss at a known world position, which projects back into that pano's pixels as a training target — no inventory, no human review, and it works on any source the labeler can enumerate. Measured yield (#102, upper bound): 0.11–0.13× the operational detections in the GSV cities and 0.32× in richmond, the one Mapillary city in the table. Richmond having the highest yield is itself the finding: the model misses more, per ramp, on the rig it has never seen.

B. Stage 1 on open imagery where a city inventory exists. Same projection Stage 1 already does, with the labeler's poses instead of Google's. Bounded by #96's location-precision gate; a fallback, not the plan.

C. Crowd validations of submitted AI labels (Vancouver on GSV at scale; Richmond on Mapillary, now live). These confirm the model's own detections, so they anchor precision but cannot recover a miss. Useful as a precision prior for A, not as a recall source.

Plan, in order

Step 1 — the pre-registered precision check (no GPU, no new labelling)

#102 says the first step is not the miner, and I agree: measure the precision of the mined set against verdicts that already exist.

On richmond (124 fully judged panos with missed-ramp marks, highest yield): for every site with ≥3 operational panos, take each judged pano within R of the site that contributed no member, project the site into that pano (bearing → x; flat-ground depression → y), and classify against the pano's verdicts inside a match window:

outcome	meaning
a reviewer missed-ramp mark inside the window	true mined positive — the model missed a ramp the reviewer saw
a verdict-true detection inside the window	already detected; an association gap, not a miss — reported separately, excluded from the precision denominator
pano judged, no mark inside the window (no_missed set or other marks only)	false mined positive — occlusion, ghost site, or out of view
an unsure missed mark	excluded, counted

Report precision with Wilson intervals, stratified by R (10 / 15 m), by n_operational_panos (3 / 4 / 5+), by projected range, and by best member confidence. Repeat on paterson and bend (GSV, judged) so rig and mechanism separate.

Proposed decision rule: ≥ 0.80 on the adjudicable subset → build the miner. 0.50–0.80 → add the visibility test #102 names as its second risk before mining (GSV depth planes, #111, for GSV; monocular depth for Mapillary). < 0.50 → drop A, fall back to B and C.

Caveat carried forward: richmond's run predates the storage floor (sidewalk-auto-labeler#28), so its sites have no sub-threshold members. The check uses operational members only; the miner itself will want the sub-floor band that newer runs store.

Step 2 — anchor the range before mining

#102's first risk: mining bakes the flat-ground scale error (#101) into the targets, systematically. For GSV the error is the camera-height constant — per-pano depth puts it at 1.11–2.50 m against the 2.6 m in geo.py (sidewalk-auto-labeler#40), and correcting it flattens the residual across every range bucket; the fix is measured and unapplied. Mapillary and Panoramax serve no depth, so they need rig-class heights (sidewalk-auto-labeler#53) checked by the ground-truth-free reprojection residual (sidewalk-auto-labeler#36). Tilt conventions are settled (sidewalk-auto-labeler#42).

Step 3 — retrain on the Run A harness

#84's Run A replicated the paper run end to end, so the training path is live. Training set = Stage 1 GSV + mined open-imagery positives. Ride-alongs, because regeneration is the moment to fix them: seam wrap in Stage 1 peak extraction (#132), the 2× input-resolution arm (#25), the canonical cropper (#113). Score on the benchmark under the protocol below and add the row to the scoreboard.

The decision this needs: evaluation protocol

Every ground-truthed Mapillary city is a benchmark split, so training on a city's full run leaks into its own score. Options:

• (a) leave-one-city-out folds — k retrains, cleanest, most expensive;
• (b) dedicated training-only cities — two or three new open-imagery runs (a city run is hours on Hyak) plus Panoramax Lyon (#159), and the ten existing splits stay untouched;
• (c) a hybrid where the judged cities stay eval and only new cities train.

My recommendation is (b): the benchmark stays comparable with every leg already on the board, and the retrain's number is a clean cross-city, cross-rig result.

Running untouched

#51 (YOLO campaign A on Tillicum) and #156 / #157 (the Fable leg) are eval-side and continue as they are. #86 stays the north star; this is its prerequisite on open imagery.

Related

#159, #102, #151, #84, #59, #54, #25, #132, #113, #111, #47, #96, #86 · sidewalk-auto-labeler#27, #36, #40, #42, #53, #20, #28

🤖 Generated with Claude Code — Fable 5.1, claude-fable-5-1

Start with docs/model_scoreboard.md and issue #102, then run the Richmond precision check using the judged panos and missed-ramp verdicts described here. Read geo.py and the related reprojection issues before anchoring range estimates. Done means the precision decision is reported, the selected open-imagery training protocol is implemented through the Run A harness, and the benchmark scoreboard gains a comparable retraining result.