ProjectSidewalk / ProjectSidewalk/RampNet
Carry the review notes and Bend's train-overlap flag into the published benchmark
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7
- Forks
- 1
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 7
Description
Two label-side gaps left over from #21, which shipped records, native, 4096x2048 and galleries (9 splits each, 11.41 GB) to rampnet-benchmark. Both are fixable with a records-only rebuild (179 KB) plus a card re-push — no imagery moves, and export_benchmark.py adopt (added in #125) makes a partial rebuild safe.
1. The reviewer's notes do not travel with the data
verdicts.json carries a top-level review_notes block (what fought the rubric, how confident the reviewer is in their own pass) and an optional per-pano note. score_validation.py prints review_notes above the numbers on purpose — so a caveat reaches whoever quotes a precision figure. The published parquet rows carry neither, and the card points back at benchmark/ in git instead.
That is exactly backwards for the audience that matters here: someone who does load_dataset(...) and never opens the repo. benchmark/README.md has asked for this since the bundle format was written — "whoever finishes #21 should make the caveats travel with the data".
Budapest is the split this exists for: its ground truth was reviewed at LOW confidence, and nothing on the Hub says so.
2. Bend's four training-overlap panoramas are unflagged
Bend is one of the paper's three training cities. An exact-id check on 2026-07-22 found 4 of its 110 reviewed panoramas in rampnet-dataset's train/val splits:
6WC0hdAYRsSAcluKSs5iRg 9kW9cxpuj7q8DMzf-ClrQQ
DJ8Zp111zu6KnMZz-0PHgQ VgWpqFkTwCIROvM0z-DkOw
The superseded scripts/build_benchmark_dataset.py kept those rows but flagged them train_overlap = true, so a consumer could score either way. scripts/export_benchmark.py — the exporter that actually published the dataset — has no such column, so the published records config cannot be filtered, and neither the card nor benchmark/README.md mentioned the overlap until now.
Measured 2026-08-18 (score_validation.py on the committed bundle with those 4 panoramas removed):
| bend | precision | recall |
|---|---|---|
| all 110 reviewed panos (published) | 0.954 | 0.758 |
| 106, overlap removed | 0.956 | 0.753 |
| unbiased 105 (published) | 0.972 | 0.738 |
| unbiased 101, overlap removed | 0.976 | 0.731 |
So this is an honesty/filterability fix, not a numbers fix — every delta is well inside the Wilson intervals (±~5 pt at these counts), and no conclusion in docs/ rests on it. It is still the kind of thing a benchmark must state rather than leave for someone to discover.
Two related gaps worth closing at the same time:
- The overlap check has not been re-run since 2026-07-22, and its id list survives only as a constant in a script nothing calls.
- No other split has been checked. Only the GSV splits can overlap at all (
bend,paterson,gainesville,sao_paulo— Mapillary ids are a different id space), and of those only Bend is a training city, so the expected answer is zero. That is a prediction, not a measurement.
Shape of the fix
build_records()gainstrain_overlap(bool, from a committed id list rather than a script constant),review_notes(per split) andnote(per pano).- Re-run the check for all four GSV splits and commit the result — including the zeros.
python scripts/export_benchmark.py adopt → build --benchmark benchmark → verify → push --message ..., then regenerate the card so therecordscolumn table documents the new fields.scripts/build_benchmark_dataset.pyandtests/test_benchmark_dataset.pyretire oncetrain_overlaplives in the real exporter.
🤖 Generated with Claude Code (claude-opus-5[1m])
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with build_records() and the adopt/build/verify flow in scripts/export_benchmark.py, then inspect benchmark/README.md and the existing Hub card generation. Re-run the overlap check for all four GSV splits and ensure records carry the stated fields and notes. Done means the records-only rebuild is verified and pushed, the card documents the fields, and the superseded script and tests are retired.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- data-engineering, documentation
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 48/100