NVIDIA / NVIDIA/Megatron-LM

[QUESTION] How to optimize parameter settings to run qwen3-30b-a3b pre-training on GB300 and fully utilize the GPU computing power

Open
#3,245 1 comment 0 reactions 1 assignee Claimed by @gautham-kollu View on GitHub
community-request module: moe question waiting-on-customer
Dominant language
Python
Stars
17.9k
Forks
4.5k
Avg merge
4d 6h
Merged PRs (30d)
271

Description

I am running the pre-training of Qwen3-30B-A3B with MXFP8 on two nodes of GB300, referencing the following resources:
Qwen3-30B-A3B configuration file: https://github.com/yanring/Megatron-MoE-ModelZoo/blob/main/model_configs/benchmarking/Qwen3-30B-A3B.yaml
NVIDIA Megatron-LM MoE implementation: https://github.com/NVIDIA/Megatron-LM/tree/dev/megatron/core/transformer/moe
I have enabled CUDA Graph and HybridEP features, but the measured single-GPU computing power is significantly lower than the 666 TFLOPS per GPU for GB300 stated in NVIDIA's official documentation (https://docs.nvidia.com/nemo/megatron-bridge/latest/performance-summary.html#system-dgx-gb300), resulting in a large gap in computing power utilization.
grouped-gemm, CUDA Graph, HybridEP and EP8 are used, are there any other optimizations the not included.? Any suggestions on parameter tuning, feature enablement, or hardware-aware optimization strategies would be greatly appreciated. Thank you!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.