Low performance of a multi-node SFT example from README.md
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
**Describe the bug**
Running the SFT multi-node example on Slurm cluster on 2 nodes with 8 H100s leads to low reported performance (in terms of TFLOPS/MFU) in v0.4.0. Similar results were observed for r0.5.0 branch. Is it expected?
Llama 3.2 1B:
📊 Training Results:
• Loss: 0.3411
• Training FLOPS: 132.47 TFLOPS (8.28 TFLOPS per rank)
• Training Model Floating Point Utilization: 0.84%
⏱️ Timing:
• Total step time: 0.33s
• policy_training: 0.32s (99.5%)
• data_processing: 0.00s (0.4%)
Llama 3.1 8B:
📊 Training Results:
• Loss: 0.2602
• Training FLOPS: 344.98 TFLOPS (21.56 TFLOPS per rank)
• Training Model Floating Point Utilization: 2.18%
⏱️ Timing:
• Total step time: 0.85s
• policy_training: 0.85s (99.8%)
• data_processing: 0.00s (0.1%)
Attaching the logs and run scripts.
[multinode_sft_nemo_rl.sh](https://github.com/user-attachments/files/24934575/multinode_sft_nemo_rl.sh)
[multinode_sft_nemo_rl_llama3.1_8b.sh](https://github.com/user-attachments/files/24934580/multinode_sft_nemo_rl_llama3.1_8b.sh)
**Steps/Code to reproduce bug**
Run `multinode_sft_nemo_rl.sh` or `multinode_sft_nemo_rl_llama3.1_8b.sh` on the login node of a Slurm cluster.
Example logs:
[llama_3.2_1b_ray-driver.log](https://github.com/user-attachments/files/24935018/llama_3.2_1b_ray-driver.log)
[llama_3.1_8b_ray-driver.log](https://github.com/user-attachments/files/24935008/llama_3.1_8b_ray-driver.log)
**Expected behavior**
Reported SFT performance is comparable frameworks (e.g. Megatron Bridge)
**Additional context**
Let me know if you need any additional information. Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.