[Baby 9B dense] Base model annealing

Base model annealing for the Prelude/baby 9B model

Plan phase 1 pretraining

For now the pretraining with stable learning rate after warm up is to train until 10T tokens without annealing. Is this desired?

Goal

Determine the annealing strategy, including having the right data prepared, config updated on oellm-autoexp and determine the budget

For discussion on the content check this document: https://docs.google.com/document/d/1wxgsDz1NTg1U8ZQ4NJri6H9jDZiaDoft8cwmDsG3dYU/edit?tab=t.0#heading=h.4hnb91ajnytx

Tasks

• Decide whether to

  • Anneal before reaching 10T tokens, so changing the final phase of the baby run
  • Anneal after 10T tokens and let baby finish
  • Anneal before reaching 10T tokens but finish the 10T baby run if budget allows)

• Decide the annealing strategy

  • Only high quality mix
  • High quality mix with increased sequence length
  • How many stages? Curriculum learning for context extension?

• Determine dataset mixture to anneal on

• How many steps annealing and what is the minimum learning rate to anneal to

• Prepare the annealing data mix on the cluster

• Prepare the annealing oellm-autoexp config on the cluster

• Check if fits within Leonardo OELLM_prod2026+OPLLM_prod2026 necessary compute budget (OPLLM_prod2026 has end date of 1 October 2026)

• Update model card