Would 2-3M training labels buy recall? Measure the scaling curve and the label ceiling before a sourcing campaign

Question

Should RampNet 2.0 chase a much larger Stage 1 training set — 2–3M labels (or 10× the panoramas) instead of today's 214,376 panos / 849,895 labels — and would that shift performance?

This issue records the analysis, the reasons to be skeptical of volume for volume's sake, and the three cheap experiments that should settle it before anyone commits to a multi-month data-sourcing campaign.

TL;DR

Volume alone probably won't move much. Volume plus new domains plus better label recall probably would. Everything we have measured says the binding constraints are (a) apparent size / input resolution, (b) the operating point, and (c) domain shift — none of which is fixed by more samples drawn from the same three GSV cities. There is also a plausible mechanism by which naive scaling makes the worst failure mode worse (see "The label ceiling" below).

But we have never actually plotted a data-scaling curve for this model, so this is inference from adjacent measurements, not a measurement. Experiments E1–E3 are the cheap way to convert it into one.

What the measurements say the bottleneck is

From docs/detection_recall_analysis.md (#25) and the model-comparison runs (docs/model_comparison.md):

Evidence	Number	Implication for "more data"
Recall by distance	0–8 m .842, 8–12 m .879, 12–18 m .812, 18–25 m .564, 25 m+ .182	Reliable to ~18 m, effectively blind past 25 m
Recall by apparent size	20–32 px .189, 32–50 px .671, 50–80 px .825, 80 px+ .876	At 4096 px wide a far ramp is a few dozen pixels. More examples do not add pixels.
Share of "misses" that were sub-threshold	~44%	Much of the gap is calibration, not vision (#54: threshold 0.55 → 0.25–0.35 buys +7–10 recall pts, free)
In-distribution gold set (1k panos, TTA, conf 0.55)	P .949 / R .873	Little in-distribution headroom left to buy
OOD degradation (clovis, 2018 GoPro Fusion)	P .914 / R .713 vs richmond .960/.765, bend .954/.758	The real loss is domain, not sample count
Resolution forecast (#25)	+0.103 recall at 2× linear res, saturating ~0.875	Needs a retrain, and needs far-field examples to learn from

So the two places with large headroom — far field and OOD imagery — are not places where "another 2M NYC-style GSV labels" obviously helps.

The label ceiling (the argument I find most concerning)

Stage 1 agreement against the manual gold set is P .9403 / R .9245. That recall figure means ~7.5% of gold-visible curb ramps are unlabeled in our training panoramas — from both pipeline misses (GPS→pixel projection, crop-model localization) and gaps in the government inventory itself.

In a heatmap-regression setup, an unlabeled ramp inside a positive panorama is not neutral: the target heatmap is zero at that location, so the loss actively pushes activations down there. Those are implicit hard negatives.

Hypothesis: Stage 1's misses are disproportionately far / small / occluded ramps — exactly the regime where projection error and the crop model degrade. If so, we have been training the model to suppress detections at precisely the range where it is now blind, and scaling to 2–3M labels from the same pipeline bakes that in harder rather than softening it.

This is a hypothesis, not a result. It is cheap to test — see E1.

Three experiments before any sourcing campaign

E1 — Is the label set the ceiling? (~1 day, no training)

Run the existing stratification tooling against the Stage 1 generated labels vs. the 1,000-pano gold set, instead of against model predictions:

• Reuse scripts/analysis/size_analysis.py and scripts/analysis/precision_by_distance.py, swapping model detections for Stage 1 output.
• Produce Stage-1 label recall stratified by distance and apparent size, on the same bins as #25.
• Compare the two curves.

Decision rule: if Stage 1's recall-by-distance curve mirrors the model's cliff, the model's far-field blindness is substantially inherited from the labels → label quality at range outranks label volume, and naive scaling is contraindicated. If Stage 1's labels are roughly flat with distance while the model's collapse, the ceiling is the model/resolution, not the data.

Needs the generated Stage 1 dataset, which is not in git (download_dataset.py, or run on makelab2) — same prerequisite as the closed #18 re-quantification.

E2 — Are we even data-limited? (~1–2 days on Hyak)

We train 1 epoch (stage_two/train.py --epochs, default 1, as in the paper). Every sample has been seen exactly once, so we do not know whether the current 214k is exhausted.

• Train 2–4 epochs on the existing dataset; track gold-set AP/recall per epoch.

Decision rule: if validation keeps improving, we are not data-starved and more epochs is ~100× cheaper than a sourcing campaign. If it plateaus while train loss keeps falling, we have saturated this distribution and new data is justified.

E3 — The actual scaling curve (~3 runs)

• Train at 25% / 50% / 100% of the current dataset (subsample by city-balanced, ramp-disjoint groups, honoring the same leakage constraints split_dataset.py enforces) and plot gold-set recall/AP vs. log(N).

Three points won't nail the exponent, but they give the slope at the current operating scale — the difference between "another 10× buys 1–2 points" and "another 10× buys 8" is exactly what we need to justify the bill. Pair with #51 (supervised YOLO baseline), which isolates architecture vs. data on the same corpus.

If we do scale: get different data, not just more

Ranked by expected value per unit of effort:

1. Open-gov ramp GPS × Mapillary imagery, in cities where both exist. Simultaneously more data and deployment-domain data — it attacks the clovis-style loss that volume cannot. RampNet 2.0's committed deployment domain is Mapillary anyway, and it sidesteps the undocumented Google endpoints the current pipeline depends on (docs/data_provenance.md).
2. Break the NYC monopoly. NYC contributes 217,680 of 276,615 government locations (~79%); Portland 45,324, Bend 13,611. Our ramp-design vocabulary is largely one city's standards. Twenty mid-size cities at 50k labels each is worth more than another NYC.
3. Repair the far-field label tail with an offline high-recall ensemble. The #39 result — OWLv2 recovers 69 of RampNet's 72 richmond misses, union recall .990, at hopeless precision (36–128 FP per recovered ramp) — is useless in production but is exactly the profile you want in an offline label generator that a precise VLM or a human then filters. This is the honest cash-out of the VLM work (#35): make 2M labels better, not merely more numerous.
4. Pair volume with the resolution retrain (#20/#25). Far-field examples only teach if the input carries the pixels — 72.4% of Richmond panos are >4096 px wide (median native 11000×5500) and get downscaled before inference. Volume × resolution is multiplicative; volume alone is not.

Diversity note: labels within a city are spatially correlated and the same ramp recurs across panoramas (split_dataset.py already has to handle this), so 3× the labels is well under 3× the independent information. Count unique ramps, not labels, when sizing a target.

Cost

Current training is 1 epoch on 214k panos at 2048×4096, >24 h on 16× L40S. A 10× dataset is ~10× that per epoch — the largest Hyak ask on the roadmap attached to the least-certain lever. E1–E3 together cost a small fraction of one such run.

Recommended ordering (recall-first)

1. #54 — drop the operating point to 0.25–0.35 (free, +7–10 recall pts) and #38 — per-ramp-across-panos sampling density. Both are config, not training.
2. E1 — Stage-1 label recall stratified by distance. Cheapest, and it can invalidate the whole scaling premise.
3. E2 / E3 — epoch ablation and the data-fraction scaling curve.
4. Higher-res retrain (#20/#25) + domain-diverse data (Mapillary, more cities) at 2–5×, with the far-field label tail repaired.
5. 10× more same-distribution GSV panos — last, and I would expect low single-digit gains for a very large bill.

Refs

#54 (operating point), #38 (per-ramp recall / sampling density), #25 (resolution), #20 (backbone/model harness), #51 (YOLO baseline — architecture vs. data), #35 (hybrid / distillation), #39 (open detectors; OWLv2 as recall oracle), #44 (benchmark-side source/geographic diversity — this issue is the training-side counterpart), #18 (closed; Stage 1 re-quantification, same data prerequisite as E1).

🤖 Generated with Claude Code

Start with scripts/analysis/size_analysis.py and scripts/analysis/precision_by_distance.py, using the generated Stage 1 dataset prerequisite described in E1. Then read stage_two/train.py and split_dataset.py for the epoch ablation and city-balanced data fractions; done means producing the requested stratified label curves and gold-set AP/recall scaling results to guide the sourcing decision.