ProjectSidewalk / ProjectSidewalk/RampNet
Rerun the collapsed YOLOv11 pano baselines with a stabilized (larger physical batch) config
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 7
- Forks
- 1
- Avg merge
- 4d 1h
- Merged PRs (30d)
- 8
Description
Parent: #51
What happened
Two of the supervised-YOLO baseline runs — the YOLO11 pano configs y11x_pano
(Slurm 37745362) and y11l_pano (37745361) — collapsed during training on Hyak:
- Both hit their epoch-1 val (~0.777 / 0.779 mAP50) — which is the COCO-pretrained
backbone's value before fine-tuning + mosaic aug engage, a pretrained artifact,
not a real score — then val mAP fell to ~0 and stayed there for 4+ epochs
(y11x_pano: four straight epochs at literal 0.000;y11l_pano: flickering ~0.02,
P≈0.94 / R≈0.03). - Training loss stayed healthy and decreasing (box 1.5→1.4, cls 1.3→1.2); no
NaN/Inf, no AMP failure in the logs. So this is a validation-side collapse, not a
gradient blow-up. - YOLO26 pano and both tiles runs were unaffected — they recover from the same
post-epoch-1 dip and climb (y26_panoreached 0.624 mAP50 by epoch 12, still rising).
Full evidence (per-epoch results.csv, resolved args.yaml, provenance) is preserved
at scripts/model_comparison/yolo_baseline/ (branch chore/yolo-baseline-record).
Leading hypothesis
Small physical batch at high input resolution → BatchNorm/EMA instability.
y11x_panoranbatch=2,y11l_panobatch=4, bothimgsz=1280.- Tiles runs (
imgsz=1024,batch=6) and YOLO26 (NMS-/DFL-free, different arch) were
fine. - Gradient accumulation (
nbs=64, already active) fixes the optimizer step but not
BN statistics, which are computed on the 2–4 physical samples. The preempt→resume
cycle on ckpt may have aggravated it.
Plan
-
Kill jobs 37745362 + 37745361 (their
best.pt/results.csv/logs are already
preserved — nothing is lost;best.ptholds only the non-reportable epoch-1 artifact). -
Rerun both v11 pano configs with a larger physical batch, in order of preference:
- (a) Lower
imgszto ~960 so more images fit → bigger batch on the same GPU.
Pass the matching--yolo-imgszat eval time (the launcher already warns about this). - (b) Constrain to a higher-VRAM ckpt GPU (h200/80G, via
--constraint) and raise
BATCH, accepting the requeue-onto-smaller-GPU risk the pinned-batch design avoids. - (c) Keep
imgsz=1280but add stabilizers: lowerlr0(0.01→0.005), lengthen
warmup_epochs(3→5), and/orfreezethe early backbone layers.
Start with (a) — cheapest, doesn't touch the schedule design.
- (a) Lower
-
A clean rerun also confirms or refutes the BN hypothesis — a scientific bonus.
Acceptance
- Val mAP50 curve is smooth / monotonic-ish (no collapse-to-zero).
- The reported number is the best-val checkpoint of a stable run, per the protocol
issue.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the preserved evidence in scripts/model_comparison/yolo_baseline/, including the resolved args.yaml, results.csv, and provenance for y11x_pano and y11l_pano; inspect the launcher’s existing yolo-imgsz and batch options. Rerun both pano configs using the lower-imgsz approach first, evaluate with the matching --yolo-imgsz, and confirm that the best-val checkpoint comes from a stable, non-collapsed curve.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 52/100