ProjectSidewalk / ProjectSidewalk/RampNet
Add the 1k manual gold-standard set as a model-comparison split (bigger, independently-labeled, un-anchored GT)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7
- Forks
- 1
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 7
Description
Motivation
The model-comparison harness (#34 / #36 / #40) scores RampNet vs the VLM/open-detector field on the richmond/bend/clovis verdict GT. That GT is deliberately deployment/OOD-focused, but it is explicitly "RampNet-anchored" — derived from a review of RampNet's own detections plus a missed-ramp pass (the docs/model_comparison.md caveat, and the bias behind #55).
We already have a bigger, independently-labeled, un-anchored gold set and aren't using it in the comparison: the 1,000-panorama Manual Ground Truth Set (3,919 curb ramps). It is:
- Fully and manually labeled from scratch by humans (not seeded by RampNet) → no RampNet-anchoring, so it sidesteps #55 entirely for this data and is fairer to the VLMs too (their recall isn't measured against a GT built from RampNet's hits).
- 4× the size of the current benchmark (1,000 vs 234) → tighter Wilson CIs.
- The paper's canonical gold standard —
README.md(§"Evaluating") says we "place more emphasis on [it]… less prone to errors… directly from a human source."
Where the data is
- Images: in
projectsidewalk/rampnet-dataseton HF (the manual-eval panos are held out from train/val, so they live in thetestsplit — see README data-leakage note). - Labels: in-repo at
manual_labels/*.txt, YOLO format (class cx cy w h, normalized), keyed by pano id.
Important caveat — it's in-distribution (GSV)
The gold set is GSV, sampled from cities including some in training (NYC/Portland/Bend; docs/data_provenance.md flags in-training cities as optimistically biased). It is held out from Stage-2 training, but it's the same imagery source + overlapping geography RampNet was built for, so RampNet has home-field advantage and its margin here will likely be larger than on OOD Mapillary. Frame it as an in-distribution reference that complements the OOD cities (richmond/clovis), not a replacement — don't over-read an in-distribution win.
What it takes (harness is split-generic)
- Fetch the 1,000 gold-set images from
projectsidewalk/rampnet-dataset(by pano id frommanual_labels/) into abenchmark/manual_gold/split (panos/git-ignored like the cities). - GT adapter: convert
manual_labelsYOLO boxes → point GT in the harness's derived-GT shape —gt_points= box centers,ignore_points= [] (nounsure),fn_confirmed= True (fully labeled ⇒ every pano recall-eligible). Same box→center reduction the harness already applies to VLMs; curb-ramp boxes are small, so center is a fine point proxy at radius 0.022. - Run the full model set (rampnet, gemini, qwen, owlv2/gdino, molmo) on the split; add a results row to
docs/model_comparison.mdwith P/R + AP + Wilson CIs. (RampNet + open detectors free; Gemini on 1,000 panos is more paid calls but within the pre-approved budget.) - Sanity-check RampNet's harness numbers against the known gold-set figures (HF card 0.949/0.873, with TTA + one-to-one matching) — expect a small offset since the harness path is no-TTA + perspective-tiled.
Notes
- Multi-city mixed set (not a single "city").
- Possible pano overlap with the existing
bendsplit (different sample; likely disjoint — confirm and note).
Refs #34, #20, #39, #45, #55.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the split-generic model-comparison harness, manual_labels/*.txt, README.md, and docs/data_provenance.md. Fetch the 1,000-panorama set into benchmark/manual_gold/, add the YOLO-to-point GT adapter, run the listed model set, and update docs/model_comparison.md with metrics, confidence intervals, and the stated in-distribution caveats.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- data, documentation, machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100