ProjectSidewalk / ProjectSidewalk/RampNet

FAIR release of rampnet-dataset + rampnet-benchmark: DOI, full cards, Croissant 1.1/GeoCroissant/RAI, evaluation protocol, leaderboard (NSF FAIROS)

Open
#150 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

documentation enhancement FAIROS
Dominant language
Python
Stars
7
Forks
1
Avg merge
4d 11h
Merged PRs (30d)
7

Description

Part of the NSF FAIROS work (tracker: ProjectSidewalk/SidewalkWebpage ProjectSidewalk/SidewalkWebpage#5053; award #2623243). Objective 3 names
"RampNet 2.0" — a DOI-versioned curb-ramp detection benchmark with fixed splits, a standard evaluation protocol,
and a leaderboard — as a Year-2 deliverable. Most of the substance already exists on Hugging Face
(projectsidewalk/rampnet-dataset: ~214K panoramas / ~850K labels, fixed train/val/test splits;
projectsidewalk/rampnet-benchmark: 9 cities across two imagery providers; the 1K manual gold set; the
v1.1-corrected-eval erratum; docs/replication.md and docs/data_provenance.md). What's missing is the FAIR
packaging. This issue is that packaging.

Tasks

  • Persistent identifier. Mint a DOI for rampnet-dataset and rampnet-benchmark (Hugging Face mints a
    DataCite DOI per revision; the card currently says "DOI: forthcoming"). Decide final repo names before
    minting — a minted repo cannot be renamed or deleted. Cite the ICCV'25 paper and the erratum in the record.
  • Dataset cards that meet the Hugging Face template fully: composition, collection process, splits and
    their rationale (city-held-out vs. random), the seam-duplicate note, known biases (city mix, imagery
    provider, capture years), intended uses, citation (dataset DOI + paper), maintenance/versioning policy
    (major version on any file change, minor on metadata-only).
  • Croissant 1.1 + GeoCroissant 1.0 + RAI metadata authored by hand (croissant.json in the repo and
    uploaded to the HF record; HF's auto-export is Croissant 1.0 only and has no geo/RAI fields). Validate with
    mlcroissant. This also satisfies the NeurIPS Datasets & Benchmarks submission requirement.
  • Evaluation protocol as code: one evaluate.py (or rampnet.eval module) that takes predictions in a
    documented format and outputs AP / P / R with the 1:1 matching rule from the erratum, plus a
    leave-one-city-out runner over rampnet-benchmark. Pin the exact split files by content hash.
  • Leaderboard: Hugging Face "private evaluator" pattern — public results dataset + Gradio leaderboard
    Space, hidden test labels for the benchmark cities, submissions via PR/dataset upload; mirror the results
    table as JSON in this repo for permanence. (Papers with Code is gone; Codabench is the fallback if code
    submissions become necessary.)
  • Carry the review notes from #127 into the published benchmark card; close #58 (gold set as split) and
    #21/#10 by pointing at the record of truth.
  • Baselines: publish the RampNet result and at least one open-weight detector baseline (#39 / #51) on the
    leaderboard at launch.

Acceptance

DOI resolves to a complete card; mlcroissant validates the metadata; evaluate.py reproduces the paper's
corrected numbers from the released predictions; the leaderboard shows ≥2 entries; the benchmark page on the
Project Sidewalk data portal links here.


🤖 Generated with Claude Code (claude-fable-5, effort: xhigh)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the existing Hugging Face dataset and benchmark cards, docs/replication.md, docs/data_provenance.md, and the v1.1-corrected-eval erratum. Then inspect the planned croissant.json and evaluation entry point (evaluate.py or rampnet.eval), including the fixed split files. Done means the DOI records, metadata validation, corrected evaluation reproduction, and leaderboard satisfy the stated acceptance checklist.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
computer-vision, data, documentation, machine-learning, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.