NVIDIA-NeMo / NVIDIA-NeMo/RL

Low performance of a multi-node SFT example from README.md

Open
#1,844 1 comment 0 reactions 1 assignee Claimed by @guyueh1 View on GitHub
bug
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

**Describe the bug**

Running the SFT multi-node example on Slurm cluster on 2 nodes with 8 H100s leads to low reported performance (in terms of TFLOPS/MFU) in v0.4.0. Similar results were observed for r0.5.0 branch. Is it expected?

Llama 3.2 1B:
📊 Training Results:
• Loss: 0.3411
• Training FLOPS: 132.47 TFLOPS (8.28 TFLOPS per rank)
• Training Model Floating Point Utilization: 0.84%

⏱️ Timing:
• Total step time: 0.33s
• policy_training: 0.32s (99.5%)
• data_processing: 0.00s (0.4%)

Llama 3.1 8B:
📊 Training Results:
• Loss: 0.2602
• Training FLOPS: 344.98 TFLOPS (21.56 TFLOPS per rank)
• Training Model Floating Point Utilization: 2.18%

⏱️ Timing:
• Total step time: 0.85s
• policy_training: 0.85s (99.8%)
• data_processing: 0.00s (0.1%)

Attaching the logs and run scripts.

[multinode_sft_nemo_rl.sh](https://github.com/user-attachments/files/24934575/multinode_sft_nemo_rl.sh)

[multinode_sft_nemo_rl_llama3.1_8b.sh](https://github.com/user-attachments/files/24934580/multinode_sft_nemo_rl_llama3.1_8b.sh)

**Steps/Code to reproduce bug**

Run `multinode_sft_nemo_rl.sh` or `multinode_sft_nemo_rl_llama3.1_8b.sh` on the login node of a Slurm cluster.

Example logs:

[llama_3.2_1b_ray-driver.log](https://github.com/user-attachments/files/24935018/llama_3.2_1b_ray-driver.log)

[llama_3.1_8b_ray-driver.log](https://github.com/user-attachments/files/24935008/llama_3.1_8b_ray-driver.log)

**Expected behavior**

Reported SFT performance is comparable frameworks (e.g. Megatron Bridge)

**Additional context**

Let me know if you need any additional information. Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.