Checklist for monitoring and reliability setup for production runs (9B baby & flagship models)

Define monitoring and reliability setup for production runs (8B baby & flagship models)

Before launching production training runs, we need to define and implement a set of monitoring, alerting, and fault-tolerance mechanisms. This covers both the 8B baby model and the flagship model trainings.

Tasks

• Set up continuous WandB log upload during training, from a tmux session on the login node or by reusing the existing sreport process. Make sure the WandB folder for each run is on a shared directory accessible to everyone.
• Define which failure conditions should trigger oellm-autoexp auto-restarts, and investigate what else can be automated from there
• Create a dedicated Mattermost channel per production run with automated notifications (job status, loss spikes, unexpected stops, SLURM job events, etc.)
• Define a babysitting schedule to assign someone to periodically check loss, LR and throughput
• Investigate fault tolerance options for GPU/node failures during training (e.g. elastic launch via torchrun) to avoid going back to the SLURM queue