SFT-to-DPO Spectral Boundary

🧠 The "Why" (Problem & Motivation)

The standard alignment pipeline is SFT → DPO, but the transition point is chosen by heuristic: "wait for the SFT loss to plateau," with no principled signal for when to switch. Under a fixed total compute budget (SFT + DPO), switching too early wastes preference data on a model that hasn't resolved enough structure; switching too late burns compute on SFT signal that DPO could have exploited better. Can we predict the right switch point and understand why it's the right one?

🚀 The Big Idea

Training consumes gradient signal from strong to weak (the LNP spectral picture, arXiv:2605.31244): rank the eNTK spectrum's directions by strength and models move from the bulk (dominant patterns: morphology, syntax) ↓ into the tail (subtle signals: multi-word negation, cross-paragraph structure) as training progresses.

• SFT follows the same bulk-to-tail dynamics as pretraining, so late in SFT the remaining signal lives in the tail.

• DPO is a paradigm shift: subtracting chosen−rejected log-probs cancels the shared signal (prompt, syntax, surface patterns) and highlights exactly the nuanced differences- the tail.

• Hypothesis: we switch from SFT to DPO to increase the signal strength.

Two cheap diagnostics, computed on held-out preference pairs (y_w, y_l) at any checkpoint, make this testable:

• R_spec = χ_pos(L_DPO) / χ_pos(L_SFT) -- the spectral-position ratio. Tests the mechanism: does the DPO gradient sit higher in the spectrum (more accessible signal) than the exhausted SFT gradient at the switch point?

• R_dir = χ_pos(L_DPO, L_SFT) / √(χ_pos(L_SFT)·χ_pos(L_DPO)) ∈ [−1, 1]-- normalized directional overlap between the SFT and DPO gradients. The practical switching signal: low overlap early (SFT hasn't built the structure DPO needs), growing as SFT resolves it, switch when it plateaus at a stable maximum.

🎯 The Dream Outcome

A validated readiness diagnostic: run R_dir/R_spec on SFT checkpoints as they come off the trainer, and read off when the model is ready for preference optimization replacing the loss-plateau heuristic with a mechanistic signal.

Validation = an empirical fixed-compute sweep (branch DPO runs off many SFT checkpoints, compare final quality) with a clean 2×2 readout:

R_spec aligns with optimum	R_dir aligns with optimum	Conclusion
✓	✓	theory supported, both diagnostics validated
✗	✓	theory weakened, but R_dir is still a practical switching tool
✓	✗	spectral picture holds, but alignment isn't the right switching signal
✗	✗	theory doesn't predict empirical behavior

---

⚙️ Prototype

Endpoint version already runs: examples/dpo_spectral_filter/ (TODO: push branch) computes both diagnostics on allenai/Olmo-3-7B-Instruct-SFT vs -DPO - no training, published checkpoints only, one H100 for a 7B model.

• Dolci run (20 pairs/domain (code and english), ~23 min): R_dir 0.59 → 0.18 SFT→DPO, R_spec 0.22–0.52.
• RewardBench scale-up (1,794 pairs, ~7 h): R_spec ≪ 1 everywhere (code 0.007 @ SFT → 0.012 @ DPO; english 0.057 → 0.013), probe-bootstrap 95% CI half-widths 2–9%; R_dir negative (code −0.27 SFT → −0.62 DPO).
• Stack: plain torch + HF, fp32 log-probs (bf16 is catastrophic for the DPO margin), chunked flat-gradient passes to fit 7B fp32 gradients on one GPU. Check out vatis!

❓ Things I'm Figuring Out / Findings

• The endpoint measurements read R_spec ≪ 1- the DPO gradient sits deeper in the tail than the SFT/CE gradient, not higher, at both checkpoints. That's the opposite sign of the naive "DPO increases signal strength" prediction (R_spec > 1 ✓ / < 1 ✗). Filter mechanism confirmed, signal-strength story needs refinement or the interesting dynamics happen between the endpoints, which is exactly what the checkpoint sweep would show.

• Two endpoints ≠ a trajectory. The switching-signal claim (R_dir plateau) needs intermediate SFT checkpoints + branched DPO from the SFT checkpoints runs i.e., our own training sweep.

• R_dir flips sign between eval sets (positive on Dolci, negative on RewardBench) and small batches get the sign wrong (+0.02 at B=10 vs −0.27 at B=984 for code@SFT). What's the minimum B for a trustworthy switching signal, and which preference distribution should the diagnostic be evaluated on?

• Heavy-tailed pair concentration (one pair = 42% of χ_net at SFT/code)- does a robust estimator (trimmed mean/median) change the picture?

• seq_len > 1024 OOMs on one H100 in the current chunked-gradient setup- longer preference pairs need sharded accumulation.

📎 References and Similar Works

• LNP decomposition / χ_pos / spectral bulk-to-tail dynamics: arXiv:2605.31244- the paper defining the observables (χ_loss, χ_net, χ_pos) both diagnostics are built from
• DPO: Rafailov et al. 2023, arXiv:2305.18290
• Code: [Todo: Push the branch]
• Checkpoints & data: allenai/Olmo-3-7B-Instruct-SFT / -DPO, allenai/Dolci-Instruct-DPO, RewardBench
• Runs/figures: W&B project

---

💬 Discussion Notes

Drop the TL;DR of decisions, pivots, and feedback here as people weigh in.