ProjectSidewalk / ProjectSidewalk/RampNet
The #140 seam fix moved three committed YOLO baseline cells, and benchmark_eval/ was never regenerated
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7
- Forks
- 1
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 7
Description
Found while re-scoring #135's Run A epoch curve after merging main into
analysis/run-b-power-135 (#138): the seam wrap from #140 changes three committed
YOLO baseline numbers, and scripts/model_comparison/yolo_baseline/benchmark_eval/
still holds the pre-fix values.
What moved
Measured over all three pano arms × all ten splits, wrap_x=False vs wrap_x=True,
same committed detections in benchmark/model_detections/ both times. Everything not
listed is identical at published precision.
| split | arm | metric | committed (pre-#140) | current scorer | note |
|---|---|---|---|---|---|
manual_gold |
y26_pano |
precision | 0.739 | 0.740 | tp/fp/fn 2895/1021/1024 → 2896/1020/1023 |
clovis |
y26_pano |
AP | 0.593 | 0.596 | counts unchanged |
richmond |
y26_pano |
AP | 0.536 | 0.537 | counts unchanged |
The manual_gold one is the clearest: one detection crossed the seam and became a
true positive instead of a false positive on one side plus a miss on the other. That
is exactly the defect #130/#140 set out to fix, so the new number is the right one —
it is only the committed artifact that is stale.
The F1 headline table in scripts/model_comparison/yolo_baseline/README.md is
entirely unaffected — every cell rounds the same. So does every y11l_pano and
y11x_pano_h200 cell, and RampNet's own numbers on every split.
docs/model_comparison.md:1310 also quotes richmond y26_pano AP as 0.536 in
prose, which is now 0.537.
The comparison is sound
Scored with wrap_x=False, the current code reproduces the committed
benchmark_eval/*.txt exactly — same tp/fp/fn, same P/R/F1/AP to three decimals,
on every row checked. So the delta is the matcher and nothing else: not a different
detection set, not a different threshold, not environment drift.
Why it slipped
#140 regenerated docs/model_comparison.md and added
tests/test_scoreboard.py::test_every_number_matches_model_comparison, which
re-derives that document's table cells on every CI run. But the YOLO pano arms are
listed there as "published, not in these tables"
(docs/model_comparison.md:57), so they sit outside that test's coverage.
benchmark_eval/ is a separate committed artifact with no equivalent guard, and
nothing failed when it went stale.
That is the generalisable part: the scoreboard test protects one document, and we
have committed numbers living in at least one other place.
Also worth correcting: the claim in score_pano's docstring
rampnet/detection_eval.py currently says:
Wrapping moves no RampNet or YOLO metric on any committed split — but it does move
the challengers.
That is not right, and it is the kind of claim someone will reasonably rely on when
deciding whether a re-score is needed. It is true for RampNet on every split and
true for every arm's F1 at the operating point, but three published YOLO cells do
move, and max-F1 — which is not in those tables but is the calibration-free column
#135 reads — moves considerably more: bend/y11x_pano_h200 by +0.0014 and
richmond/y26_pano by +0.0030.
Suggested fix
- Regenerate
benchmark_eval/from the committed detections (no GPU needed — it is a
re-score of a cached detection set) and commit the three changed cells. - Fix the
docs/model_comparison.md:1310prose figure. - Narrow the docstring claim to what was actually measured.
- Extend the scoreboard test, or add a sibling, so
benchmark_eval/is re-derived in
CI too — otherwise the next scorer change does this again silently.
Not doing any of it in #138, which only needed the Run A epoch curve re-scored; that PR
records the measurement and points here.
🤖 Generated with Claude Code (claude-opus-5[1m])
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the committed detections, the current scorer, and scripts/model_comparison/yolo_baseline/benchmark_eval/; compare the regenerated values with the listed stale cells. Then inspect docs/model_comparison.md:1310, rampnet/detection_eval.py, and tests/test_scoreboard.py. Done means the artifacts and prose are corrected, the docstring claim is accurate, and CI checks benchmark_eval as well.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- documentation, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100