ProjectSidewalk / ProjectSidewalk/RampNet

Add Clovis validation split to the HF benchmark dataset

Open
#49 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7
Forks
1
Avg merge
4d 11h
Merged PRs (30d)
7

Description

Add the Clovis split to the published HF benchmark

The Clovis validation split landed in gt/clovis-benchmark (branch, PR pending):
benchmark/clovis/ with 125 GT panos and verdicts.json. It scores P 0.889 /
R 0.650
on the unbiased subset (P 0.914 / R 0.713 across all 125) — a second
out-of-distribution Mapillary city alongside Richmond, and the weakest split so far.

scripts/build_benchmark_dataset.py currently packs only bend + richmond. To include
Clovis:

  • Add "clovis": "mapillary" to CITY_IMAGERY.
  • Training-overlap check: Clovis is Mapillary (unseen imagery source), so like
    Richmond it should have zero overlap with projectsidewalk/rampnet-dataset
    train/val. Confirm and document (no LEAKED_* set expected).
  • Verify the reload-and-rescore step reproduces P 0.889 / R 0.650 for the split.
  • Update the dataset card: Clovis is an out-of-distribution deployment city whose
    lower recall is explained by imagery — it is 100% GoPro Fusion (soft 2018
    consumer 360), vs Richmond's mostly-pro NCTECH iSTAR Pulsar + GoPro Max.
Opportunity: expose camera provenance as a dataset column

The records now carry full Mapillary provenance — camera_make / camera_model /
camera_type plus a verbatim source_metadata dump (added in the auto-labeler,
sidewalk-auto-labeler export-benchmark-verify). Since camera model is the field that
actually predicts image quality across cities (quality_score is compressed near the top
and resolution barely varies), consider adding camera_model to the dataset FEATURES
so the OOD story is analyzable directly from the published benchmark.

Related: #21 (HF benchmark), #26 (GT tool), and the Clovis split PR.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with scripts/build_benchmark_dataset.py, especially CITY_IMAGERY and the dataset FEATURES, then inspect the reload-and-rescore workflow. Add the Clovis split, verify its train/val overlap and reported precision/recall, and update the dataset card with its OOD and camera context; determine whether the camera_model opportunity is in scope.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.