ProjectSidewalk / ProjectSidewalk/RampNet

Carry the review notes and Bend's train-overlap flag into the published benchmark

Open
#127 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
7
Forks
1
Avg merge
4d 11h
Merged PRs (30d)
7

Description

Two label-side gaps left over from #21, which shipped records, native, 4096x2048 and galleries (9 splits each, 11.41 GB) to rampnet-benchmark. Both are fixable with a records-only rebuild (179 KB) plus a card re-push — no imagery moves, and export_benchmark.py adopt (added in #125) makes a partial rebuild safe.

1. The reviewer's notes do not travel with the data

verdicts.json carries a top-level review_notes block (what fought the rubric, how confident the reviewer is in their own pass) and an optional per-pano note. score_validation.py prints review_notes above the numbers on purpose — so a caveat reaches whoever quotes a precision figure. The published parquet rows carry neither, and the card points back at benchmark/ in git instead.

That is exactly backwards for the audience that matters here: someone who does load_dataset(...) and never opens the repo. benchmark/README.md has asked for this since the bundle format was written — "whoever finishes #21 should make the caveats travel with the data".

Budapest is the split this exists for: its ground truth was reviewed at LOW confidence, and nothing on the Hub says so.

2. Bend's four training-overlap panoramas are unflagged

Bend is one of the paper's three training cities. An exact-id check on 2026-07-22 found 4 of its 110 reviewed panoramas in rampnet-dataset's train/val splits:

6WC0hdAYRsSAcluKSs5iRg  9kW9cxpuj7q8DMzf-ClrQQ
DJ8Zp111zu6KnMZz-0PHgQ  VgWpqFkTwCIROvM0z-DkOw

The superseded scripts/build_benchmark_dataset.py kept those rows but flagged them train_overlap = true, so a consumer could score either way. scripts/export_benchmark.py — the exporter that actually published the dataset — has no such column, so the published records config cannot be filtered, and neither the card nor benchmark/README.md mentioned the overlap until now.

Measured 2026-08-18 (score_validation.py on the committed bundle with those 4 panoramas removed):

bend precision recall
all 110 reviewed panos (published) 0.954 0.758
106, overlap removed 0.956 0.753
unbiased 105 (published) 0.972 0.738
unbiased 101, overlap removed 0.976 0.731

So this is an honesty/filterability fix, not a numbers fix — every delta is well inside the Wilson intervals (±~5 pt at these counts), and no conclusion in docs/ rests on it. It is still the kind of thing a benchmark must state rather than leave for someone to discover.

Two related gaps worth closing at the same time:

  • The overlap check has not been re-run since 2026-07-22, and its id list survives only as a constant in a script nothing calls.
  • No other split has been checked. Only the GSV splits can overlap at all (bend, paterson, gainesville, sao_paulo — Mapillary ids are a different id space), and of those only Bend is a training city, so the expected answer is zero. That is a prediction, not a measurement.

Shape of the fix

  • build_records() gains train_overlap (bool, from a committed id list rather than a script constant), review_notes (per split) and note (per pano).
  • Re-run the check for all four GSV splits and commit the result — including the zeros.
  • python scripts/export_benchmark.py adopt → build --benchmark benchmark → verify → push --message ..., then regenerate the card so the records column table documents the new fields.
  • scripts/build_benchmark_dataset.py and tests/test_benchmark_dataset.py retire once train_overlap lives in the real exporter.

🤖 Generated with Claude Code (claude-opus-5[1m])

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with build_records() and the adopt/build/verify flow in scripts/export_benchmark.py, then inspect benchmark/README.md and the existing Hub card generation. Re-run the overlap check for all four GSV splits and ensure records carry the stated fields and notes. Done means the records-only rebuild is verified and pushed, the card documents the fields, and the superseded script and tests are retired.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
data-engineering, documentation
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.