Early stopping may cut training short: val/f1 ties don't reset patience, LR schedule never matures

Observed (v4.1.0 retrain, 2026-08-14)

Training the ViT-DINOv2 finetune on pyro-dataset v4.1.0 stopped after 8 of 30 epochs, keeping the epoch-2 checkpoint:

Why this is suspect

• Early stopping and checkpointing monitor val/f1 with min_delta=0, and a tie counts as no improvement. With only ~300 val tubes, F1 is quantized in ~0.003 steps, so plateaus-with-ties are common and patience burns through them. Epoch 5 tied epoch 2's F1 exactly while val_loss was still improving (0.1132, the run's best), yet the counter kept running.
• The cosine-with-warmup LR schedule is laid out over max_epochs: 30; stopping at epoch 7 means the model only ever trained in the high-LR region.
• Epochs cost ~70 s on a 4070 Ti Super — a larger patience is nearly free.

Counterpoints

• The same config produced the current production model, and the real gate is the downstream protocol eval (sequence-level), not tube-level F1.
• Not changed in the v4.1.0 retrain PR on purpose: that PR keeps training config identical so the old-vs-new comparison isolates the dataset change.

Suggested directions (pick after a small sweep)

• Monitor val/loss (smoother, not quantized) for early stop, keep val/f1 for checkpoint selection — or monitor both with patience on loss.
• And/or raise early_stop_patience (e.g. 8–10) so the schedule can mature.
• Sanity-check any change against the packaged-model protocol eval, not just tube-level metrics.

Config lives in train/params.yaml (_train_defaults.early_stop_patience, currently 5); callbacks in train/src/temporal_model/train/train.py.

Start with train/params.yaml to inspect _train_defaults.early_stop_patience and then read the callbacks in train/src/temporal_model/train/train.py. Run a small comparison of the proposed monitoring or patience changes, and check completion against the packaged-model protocol eval as well as tube-level metrics.