[32B Dense] Debug 64k increasing loss

While training the 32B dense flag model we encountered an increasing loss around 64k:
See:

From: https://wandb.ai/openeurollm-project/oellm_32b_dense
For live debug session: https://github.com/OpenEuroLLM/oellm-autoexp/blob/prod/oellm_32b_dense/docs/64k-debug/DEBUG.md

The Problem:

The training loss was going down like it should. Then around iteration 65,000
it turned around and started going up, and it keeps getting worse.

• Run: oellm_32b_dense_prod_dataopt5_gbs4096_lr3e-4
• Last job: 1537344 →
  wandb,
  stopped 2026-08-30 22:43 at iteration 75,125
• Training loss: 1.5434 at ~65,000 → 1.587 at 75,125

This is not a wobble. It is a steady climb that is speeding up — all of it visible
in job 1537344
(wandb, which
covers 66,626 → 75,125, i.e. the whole broken stretch):

Why this is a big deal: we are only 8.4% into a 15T-token run
(75,125 of 894,000 iterations, ~1.26T tokens). If the model gets worse the longer
we train, the other 92% cannot be run this way. Rough cost so far: ~11,000
iterations spent making it worse ≈ 185B tokens ≈ 27,500 GPU-h.

How we found the exact spot

Fitting the healthy part of the curve (12k–34k) and extrapolating, the run stays
on track until ~66,000 and then breaks away: +0.012 at 68k, +0.024 at 72k,
+0.038 at 74k. The break is at ~66,625.

Held-out scores confirm the model really got worse (lower = better):

Normally later = better. Here it flips after 64,000. That flip is the bug.

This could be due to a variety of things

• FP8
• NaN weights/activations
• Switching containers
• Turning on inter-document attention masking
• Data quality
• RMSNorm off-zero gamma