ProjectSidewalk / ProjectSidewalk/RampNet

Add clovis as a third benchmark city; pursue source/geographic diversity

Open
#44 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
7
Forks
1
Avg merge
4d 11h
Merged PRs (30d)
7

Description

Motivation

The comparison rests on two cities — richmond (Mapillary 360) and bend (GSV). That's thin for a "RampNet wins" generalization claim, and it can't distinguish "RampNet is better everywhere" from "RampNet is tuned to these two US-suburban streetscapes." More cities tighten the Wilson CIs and stress-test both the lead and RampNet itself.

Immediate: finish clovis

benchmark/clovis/ is already staged — 125 Mapillary panos (Clovis, NM), records.jsonl + panos/ + gallery/ + index.csv — but has no verdicts.json yet, so it isn't scored and doesn't appear in the results. Remaining:

  • GT-verify the sample with scripts/gt_gallery.py benchmark/clovisverdicts.json (in flight; gt_gallery.py currently has uncommitted working changes).
  • One cluster run of the model set → a third results row in docs/model_comparison.md.
  • Add the split to the HF benchmark dataset (#21) and the README "Current splits" table.

Beyond clovis: real diversity

clovis is Mapillary again, so it doesn't add source diversity (still GSV-vs-Mapillary). A genuinely harder generalization test wants a different source / geography / camera: a dense urban core, a non-US city, or a different capture rig. The two-repo split (sidewalk-auto-labeler enumerates + detects, RampNet scores) makes adding a city cheap — a split just needs records.jsonl + verdicts.json + panos/.

Bonus

clovis can double as the dev split for the VLM prompt study (companion issue), keeping prompt tuning off the richmond/bend test set.

Refs #20, #21; docs/model_comparison.md Next increments #2.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with benchmark/clovis/ and run scripts/gt_gallery.py benchmark/clovis to produce the missing verdicts.json, checking the script's current working changes first. Then run one model cluster, add the Clovis row to docs/model_comparison.md, update the HF dataset and README Current splits table, and confirm the split appears in results.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
data, documentation, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.