[32B Dense] Babysitting instructions

Babysitting instructions for 32B model

Software setup

1. Get access to Jupiter and the corresponding Strategic allocation e-sta-openeurollm, Extreme Scale budget e-ext-2025e02-108 and datasets datasets
2. Clone oellm-autoexp: git clone https://github.com/OpenEuroLLM/oellm-autoexp.git --recurse-submodules
3. Checkout to pretraining branch: git checkout prod/oellm_32b_dense
4. If no uv is installed, follow: https://github.com/OpenEuroLLM/oellm-autoexp#quickstart-using-pre-configured-enviroment-and-default-values
5. Run a test job that does not interfere with the production job: uv run python scripts/run_autoexp.py --config-name experiments/...
6. Check if you have permissions to write files:

getfacl /e/project1/e-sta-openeurollm/production_training/
getfacl /e/project1/e-sta-openeurollm/pre_production_training/
getfacl /e/scratch/e-sta-openeurollm/production_training/
getfacl /e/scratch/e-sta-openeurollm/pre_production_training/

8. Set up W&B in .venv: uv pip install wandb then wandb login

Start a new training job

1. Pull latest changes from prod/oellm_32b_dense
2. Run in oellm-exp root:

uv run python scripts/run_autoexp.py --config-name experiments/oellm_32b_dense/oellm_32b_dense_production

All necessary files including logs, checkpoints, data, wandb/tensorboard are stored at path:
/e/project1/e-sta-openeurollm/production_training/oellm_32b_dense_prod_gbs4096_lr3e-4

To exclude nodes, add them to the autoexclusion file at
/e/project1/e-sta-openeurollm/production_training/jupiter_exclude_nodes.txt

[!NOTE]
In case you will start the new training job with auto_restart, this has to be performed over tmux (persistent connection from your local to login node of Leonardo) as the daemon spawning new jobs will work on a live terminal

Babysitting duties

Schedule here https://docs.google.com/spreadsheets/d/1uYidr_VgjRBXxdYZ4b-8UXQmDJFv_jpCMtsDF_Br5Ew/edit?gid=2087395347#gid=2087395347

• Check running jobs: ...

• Monitor W&B for loss spikes and throughput (tokens/s/GPU) spikes at: https://wandb.ai/openeurollm-project/...

• Automatic W&B sync should happen already: https://github.com/OpenEuroLLM/Taskboard/issues/340

• Monitoring tool: https://github.com/OpenEuroLLM/oellm-monitoring