ProjectSidewalk / ProjectSidewalk/RampNet

Rerun the collapsed YOLOv11 pano baselines with a stabilized (larger physical batch) config

Open
#70 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Python
Stars
7
Forks
1
Avg merge
4d 1h
Merged PRs (30d)
8

Description

Parent: #51

What happened

Two of the supervised-YOLO baseline runs — the YOLO11 pano configs y11x_pano
(Slurm 37745362) and y11l_pano (37745361) — collapsed during training on Hyak:

  • Both hit their epoch-1 val (~0.777 / 0.779 mAP50) — which is the COCO-pretrained
    backbone's value before fine-tuning + mosaic aug engage, a pretrained artifact,
    not a real score
    — then val mAP fell to ~0 and stayed there for 4+ epochs
    (y11x_pano: four straight epochs at literal 0.000; y11l_pano: flickering ~0.02,
    P≈0.94 / R≈0.03).
  • Training loss stayed healthy and decreasing (box 1.5→1.4, cls 1.3→1.2); no
    NaN/Inf, no AMP failure
    in the logs. So this is a validation-side collapse, not a
    gradient blow-up.
  • YOLO26 pano and both tiles runs were unaffected — they recover from the same
    post-epoch-1 dip and climb (y26_pano reached 0.624 mAP50 by epoch 12, still rising).

Full evidence (per-epoch results.csv, resolved args.yaml, provenance) is preserved
at scripts/model_comparison/yolo_baseline/ (branch chore/yolo-baseline-record).

Leading hypothesis

Small physical batch at high input resolution → BatchNorm/EMA instability.

  • y11x_pano ran batch=2, y11l_pano batch=4, both imgsz=1280.
  • Tiles runs (imgsz=1024, batch=6) and YOLO26 (NMS-/DFL-free, different arch) were
    fine.
  • Gradient accumulation (nbs=64, already active) fixes the optimizer step but not
    BN statistics, which are computed on the 2–4 physical samples. The preempt→resume
    cycle on ckpt may have aggravated it.

Plan

  1. Kill jobs 37745362 + 37745361 (their best.pt/results.csv/logs are already
    preserved — nothing is lost; best.pt holds only the non-reportable epoch-1 artifact).

  2. Rerun both v11 pano configs with a larger physical batch, in order of preference:

    • (a) Lower imgsz to ~960 so more images fit → bigger batch on the same GPU.
      Pass the matching --yolo-imgsz at eval time (the launcher already warns about this).
    • (b) Constrain to a higher-VRAM ckpt GPU (h200/80G, via --constraint) and raise
      BATCH, accepting the requeue-onto-smaller-GPU risk the pinned-batch design avoids.
    • (c) Keep imgsz=1280 but add stabilizers: lower lr0 (0.01→0.005), lengthen
      warmup_epochs (3→5), and/or freeze the early backbone layers.

    Start with (a) — cheapest, doesn't touch the schedule design.

  3. A clean rerun also confirms or refutes the BN hypothesis — a scientific bonus.

Acceptance

  • Val mAP50 curve is smooth / monotonic-ish (no collapse-to-zero).
  • The reported number is the best-val checkpoint of a stable run, per the protocol
    issue.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the preserved evidence in scripts/model_comparison/yolo_baseline/, including the resolved args.yaml, results.csv, and provenance for y11x_pano and y11l_pano; inspect the launcher’s existing yolo-imgsz and batch options. Rerun both pano configs using the lower-imgsz approach first, evaluate with the matching --yolo-imgsz, and confirm that the best-val checkpoint comes from a stable, non-collapsed curve.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
computer-vision, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.