Anneal-matched 0.5-epoch anchor for the Olmo 3.5 SFT mixture search (KDA 1.2B, Dolci-Think, seq 65536)

Claude Code · bridge-cse session (Abhishek's #1853/#1859 anchor work) — agent-written

Canonical handle for the **anneal-matched 0.5-epoch anchor** of the Olmo 3.5 SFT mixture search (#1853). Ledger: H006 in `olmo35-sft-search`. Predecessor runs: #1854 (32K, 0.5 ep), #1859 (64K, 1 ep).

## Identity

| | |
|---|---|
| short name | `anchor-0.5ep-annealed` (W&B `w5adembs`) |
| Beaker | https://beaker.org/ex/01M1JMPGEKPAHGK1YSAGTDYSAH |
| durable checkpoints | `/weka/oe-adapt-default/abhishekr/checkpoints/olmoe3-kda-1.2b-dolci-think-sft-65536/anchor-0.5ep-annealed-w5adembs/` — `step11768` (DCP, **the anchor**), `hf_step11768`, `hf_step5884`; README inside |
| scratch original | `deletable_checkpoint_states/w5adembs/` (expendable) |

## Recipe

OLMoE3 latent-KDA 1.2B (18.5B total) LC base `cx8-samebatch/step63802` → Dolci-Think-SFT, olmo123 template, seq 65536 packed, intra-document masking, LR 2.5e-5, global batch 1,048,576 tokens, **11,768 steps = 0.5 epoch (12.3B tokens) with its own LinearWithWarmup schedule → LR reaches 0 at step 11,768.** 1×8 B300 (grad_accum 2), 13.36h, ~107 GPU-h, exit 0, never preempted. Same base, data order (seed 34521), tokens and LR as #1859's step11768 — only the schedule horizon differs.

## Why it exists

The previous anchor, #1859's step11768, is a *mid-run* checkpoint of a 23,607-step run: the scheduler is `LinearWithWarmup` with `alpha_f=0`, so it sat at ~52% of peak LR, never annealed. Mixture-search arms trained as dedicated 11,768-step runs anneal fully, so every arm would have had a free edge over that anchor. This run matches the arms' schedule. The pair (annealed vs un-annealed, same tokens) also measures the annealing effect directly, and annealed-65536 vs #1854's annealed-32768 finally isolates the sequence-window effect.

## Status

- [x] trained, converted (`-s 65536`, verified), durable copy (276 GB)
- [x] 12-task battery + RULER×4 on `hf_step11768` and `hf_step5884` (24/24 exit 0, third wave; two earlier waves lost to an eval-harness dependency drift — `huggingface-hub` → 0.36.2 at job time; fix = trailing `transformers==5.14.1,huggingface-hub==1.16.1` install steps in `eval_kda_mc.sh`)
- [x] paired comparison vs #1859 step11768 (annealing effect: negligible, H006 **refuted**) and vs #1854 step11768 (window effect: −1.5pp in-window ns, +30.9pp at 65536); RULER@65536 differs −20.8pp between the two anchors and the eval reproduces exactly → **training-side variance ≥20pp at 65536**, noise floor needed before any 65536 arm claim
- [x] adopted as the #1853 anchor (Abhishek, 2026-09-03); second seed funded for the 65536 noise floor (H001)