Qwen3 dense 8B training scale up tp 256 nodes on MN5

In June it is planned to pretrain the Qwen3 dense 8B models on MN5 with 256 nodes. To that end, we want to have the highest throughput at 256 nodes.

TODOs:

• Ask for the correct config for the Qwen3 dense 8B. Which will be the GBS for each number of nodes for the mid-size model (multilingual scaling laws?)?
• Ask for the final dataset, how it will be transfered and when. May be important to test throughput with the final dataset.
• Make sure the correct architecture is deployed in each oellm-autoexp in MN5 in this path: /gpfs/projects/ehpc533/users/$USER/oellm-autoexp
• Test container to use in this setup. Options: nemo_2507, megatron_qwen35 (megatron_core-0.18.0+251c6e9de)
• Test maximum micro batch size with each GBS.
• Test if recomputation can help.
• Weak scaling plots, with up to 256 nodes (1024 GPUs).
• Test FSDP.
• Test num_workers to avoid failure of GPFS ulimit with longer runs.
• Detect bottleneck at each node count. Ideally, we want the GPU to be the bottleneck.

Start by confirming the Qwen3 dense 8B configuration, final dataset, and container choice for MN5, then inspect the deployment path /gpfs/projects/ehpc533/users/$USER/oellm-autoexp. Work through micro-batch, recomputation, FSDP, worker-count, and bottleneck tests, and finish with weak-scaling plots through 256 nodes (1024 GPUs).