[QUESTION] How to optimize parameter settings to run qwen3-30b-a3b pre-training on GB300 and fully utilize the GPU computing power
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
I am running the pre-training of Qwen3-30B-A3B with MXFP8 on two nodes of GB300, referencing the following resources:
Qwen3-30B-A3B configuration file: https://github.com/yanring/Megatron-MoE-ModelZoo/blob/main/model_configs/benchmarking/Qwen3-30B-A3B.yaml
NVIDIA Megatron-LM MoE implementation: https://github.com/NVIDIA/Megatron-LM/tree/dev/megatron/core/transformer/moe
I have enabled CUDA Graph and HybridEP features, but the measured single-GPU computing power is significantly lower than the 666 TFLOPS per GPU for GB300 stated in NVIDIA's official documentation (https://docs.nvidia.com/nemo/megatron-bridge/latest/performance-summary.html#system-dgx-gb300), resulting in a large gap in computing power utilization.
grouped-gemm, CUDA Graph, HybridEP and EP8 are used, are there any other optimizations the not included.? Any suggestions on parameter tuning, feature enablement, or hardware-aware optimization strategies would be greatly appreciated. Thank you!
Contributor guide
Assessment
This issue has not been assessed yet.