Some datasets in Phase 3 pretraining are repeated Thousands of Times?

Hi @eliebak @loubnabnl, I'm trying to reproduce phase 3 (LR decay). I noticed that four datasets in Dolmino are extremely small. To reach the required number of tokens, we'd need to repeat `tiny-gsm-mind-problem-solving` for ~1,595 epochs and `tiny-gsm-mind-2students` for ~1,750 epochs. Is this expected? Thanks for your help!

Phase3 Dataset | Tokens_Have | Tokens_Needed | Epoch
-- | -- | -- | --
tiny-gsm-mind-problem-solving | 2.32 M | 3.69 B | 1595.31
tiny-gsm-mind-2students | 2.11 M | 3.69 B | 1750.56
dolmino_math_synth_gsm_gsm8k | 1.08 M | 493 M | 457.11
dolmino_math_synth_basic | 11.3 M | 246 M | 21.81

The issue provides no file, test, or entry point. Start by locating the Phase 3 pretraining dataset configuration and checking how token targets and repeat counts are calculated for the four named datasets; done means confirming whether those epoch counts are expected and documenting or correcting the configuration.

Written by the indexing model from the issue text.