ProjectSidewalk / ProjectSidewalk/RampNet
FAIR release of rampnet-dataset + rampnet-benchmark: DOI, full cards, Croissant 1.1/GeoCroissant/RAI, evaluation protocol, leaderboard (NSF FAIROS)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7
- Forks
- 1
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 7
Description
Part of the NSF FAIROS work (tracker: ProjectSidewalk/SidewalkWebpage ProjectSidewalk/SidewalkWebpage#5053; award #2623243). Objective 3 names
"RampNet 2.0" — a DOI-versioned curb-ramp detection benchmark with fixed splits, a standard evaluation protocol,
and a leaderboard — as a Year-2 deliverable. Most of the substance already exists on Hugging Face
(projectsidewalk/rampnet-dataset: ~214K panoramas / ~850K labels, fixed train/val/test splits;
projectsidewalk/rampnet-benchmark: 9 cities across two imagery providers; the 1K manual gold set; the
v1.1-corrected-eval erratum; docs/replication.md and docs/data_provenance.md). What's missing is the FAIR
packaging. This issue is that packaging.
Tasks
- Persistent identifier. Mint a DOI for
rampnet-datasetandrampnet-benchmark(Hugging Face mints a
DataCite DOI per revision; the card currently says "DOI: forthcoming"). Decide final repo names before
minting — a minted repo cannot be renamed or deleted. Cite the ICCV'25 paper and the erratum in the record. - Dataset cards that meet the Hugging Face template fully: composition, collection process, splits and
their rationale (city-held-out vs. random), the seam-duplicate note, known biases (city mix, imagery
provider, capture years), intended uses, citation (dataset DOI + paper), maintenance/versioning policy
(major version on any file change, minor on metadata-only). - Croissant 1.1 + GeoCroissant 1.0 + RAI metadata authored by hand (
croissant.jsonin the repo and
uploaded to the HF record; HF's auto-export is Croissant 1.0 only and has no geo/RAI fields). Validate with
mlcroissant. This also satisfies the NeurIPS Datasets & Benchmarks submission requirement. - Evaluation protocol as code: one
evaluate.py(orrampnet.evalmodule) that takes predictions in a
documented format and outputs AP / P / R with the 1:1 matching rule from the erratum, plus a
leave-one-city-out runner overrampnet-benchmark. Pin the exact split files by content hash. - Leaderboard: Hugging Face "private evaluator" pattern — public results dataset + Gradio leaderboard
Space, hidden test labels for the benchmark cities, submissions via PR/dataset upload; mirror the results
table as JSON in this repo for permanence. (Papers with Code is gone; Codabench is the fallback if code
submissions become necessary.) - Carry the review notes from #127 into the published benchmark card; close #58 (gold set as split) and
#21/#10 by pointing at the record of truth. - Baselines: publish the RampNet result and at least one open-weight detector baseline (#39 / #51) on the
leaderboard at launch.
Acceptance
DOI resolves to a complete card; mlcroissant validates the metadata; evaluate.py reproduces the paper's
corrected numbers from the released predictions; the leaderboard shows ≥2 entries; the benchmark page on the
Project Sidewalk data portal links here.
🤖 Generated with Claude Code (claude-fable-5, effort: xhigh)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the existing Hugging Face dataset and benchmark cards, docs/replication.md, docs/data_provenance.md, and the v1.1-corrected-eval erratum. Then inspect the planned croissant.json and evaluation entry point (evaluate.py or rampnet.eval), including the fixed split files. Done means the DOI records, metadata validation, corrected evaluation reproduction, and leaderboard satisfy the stated acceptance checklist.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- computer-vision, data, documentation, machine-learning, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 25/100