ProjectSidewalk / ProjectSidewalk/RampNet

Add the 1k manual gold-standard set as a model-comparison split (bigger, independently-labeled, un-anchored GT)

Open
#58 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
7
Forks
1
Avg merge
4d 11h
Merged PRs (30d)
7

Description

Motivation

The model-comparison harness (#34 / #36 / #40) scores RampNet vs the VLM/open-detector field on the richmond/bend/clovis verdict GT. That GT is deliberately deployment/OOD-focused, but it is explicitly "RampNet-anchored" — derived from a review of RampNet's own detections plus a missed-ramp pass (the docs/model_comparison.md caveat, and the bias behind #55).

We already have a bigger, independently-labeled, un-anchored gold set and aren't using it in the comparison: the 1,000-panorama Manual Ground Truth Set (3,919 curb ramps). It is:

  • Fully and manually labeled from scratch by humans (not seeded by RampNet) → no RampNet-anchoring, so it sidesteps #55 entirely for this data and is fairer to the VLMs too (their recall isn't measured against a GT built from RampNet's hits).
  • 4× the size of the current benchmark (1,000 vs 234) → tighter Wilson CIs.
  • The paper's canonical gold standardREADME.md (§"Evaluating") says we "place more emphasis on [it]… less prone to errors… directly from a human source."

Where the data is

  • Images: in projectsidewalk/rampnet-dataset on HF (the manual-eval panos are held out from train/val, so they live in the test split — see README data-leakage note).
  • Labels: in-repo at manual_labels/*.txt, YOLO format (class cx cy w h, normalized), keyed by pano id.

Important caveat — it's in-distribution (GSV)

The gold set is GSV, sampled from cities including some in training (NYC/Portland/Bend; docs/data_provenance.md flags in-training cities as optimistically biased). It is held out from Stage-2 training, but it's the same imagery source + overlapping geography RampNet was built for, so RampNet has home-field advantage and its margin here will likely be larger than on OOD Mapillary. Frame it as an in-distribution reference that complements the OOD cities (richmond/clovis), not a replacement — don't over-read an in-distribution win.

What it takes (harness is split-generic)

  • Fetch the 1,000 gold-set images from projectsidewalk/rampnet-dataset (by pano id from manual_labels/) into a benchmark/manual_gold/ split (panos/ git-ignored like the cities).
  • GT adapter: convert manual_labels YOLO boxes → point GT in the harness's derived-GT shape — gt_points = box centers, ignore_points = [] (no unsure), fn_confirmed = True (fully labeled ⇒ every pano recall-eligible). Same box→center reduction the harness already applies to VLMs; curb-ramp boxes are small, so center is a fine point proxy at radius 0.022.
  • Run the full model set (rampnet, gemini, qwen, owlv2/gdino, molmo) on the split; add a results row to docs/model_comparison.md with P/R + AP + Wilson CIs. (RampNet + open detectors free; Gemini on 1,000 panos is more paid calls but within the pre-approved budget.)
  • Sanity-check RampNet's harness numbers against the known gold-set figures (HF card 0.949/0.873, with TTA + one-to-one matching) — expect a small offset since the harness path is no-TTA + perspective-tiled.

Notes

  • Multi-city mixed set (not a single "city").
  • Possible pano overlap with the existing bend split (different sample; likely disjoint — confirm and note).

Refs #34, #20, #39, #45, #55.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the split-generic model-comparison harness, manual_labels/*.txt, README.md, and docs/data_provenance.md. Fetch the 1,000-panorama set into benchmark/manual_gold/, add the YOLO-to-point GT adapter, run the listed model set, and update docs/model_comparison.md with metrics, confidence intervals, and the stated in-distribution caveats.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
data, documentation, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.