ProjectSidewalk / ProjectSidewalk/RampNet

Re-run and publish the Vistas 1024-parity and A40-control detections (#126, #137)

Open
#163 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
7
Forks
1
Avg merge
4d 11h
Merged PRs (30d)
7

Description

The Vistas resolution-parity result in #137 (docs/model_comparison.md, the 1024-input parity arm and the same-env 384 A40 control) rests on detections that no longer exist anywhere. The only copy was a private .model_cache shard in a scratch worktree on makelab2; a search on 2026-09-17 found no registered worktree under /homes/gws/jonf/RampNet, no shard written 2026-08-17..20, and none of the parity run's shard names (compare.cache_key over the 1024 signature) under /homes/gws/jonf, /tmp, /var, the root filesystem or the lab mounts. The doc states this beside the numbers; #137 was merged with the gap stated rather than held.

The fix is cheap on GPU — both A40 arms re-run in about 3m38s on makelab2 — but publishing the result into benchmark/model_detections/ runs into three things the roster's own rules leave open, and they need a decision before the re-run PR:

  1. A pinned 1024 leg forces the published 384 file to be renamed too (test_every_leg_of_a_pinned_model_is_qualified), and the pin-naming rule expects a scalar.
  2. The 384 control shares the published run's cache key by design, so it cannot be a pinned leg as the roster stands.
  3. scripts/analysis/export_model_cache.py does not take --vistas-input-size.

Plan: settle 1–3 in rampnet/roster.py / the exporter, re-run both arms on makelab2, publish, and update the five doc sections that quote the parity numbers. The two tests that pin the pre-fix cascade state on purpose (tests/test_cascade_gate.py) will need the same treatment if the re-run does not reproduce the old numbers exactly — and if it does not, that is a result to record, not a discrepancy to smooth over.

🤖 Generated with Claude Code — Opus 5 (1M context), claude-opus-5[1m]

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading rampnet/roster.py and scripts/analysis/export_model_cache.py, then inspect the relevant tests in tests/test_cascade_gate.py and the roster qualification tests. Run the existing tests before deciding how the pinned 1024 leg, 384 control, and input-size export should be handled. Done means both A40 arms are rerun, detections are published under benchmark/model_detections/, and the five documented parity sections reflect the recorded results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, documentation, machine-learning, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.