Qwen3 235B-A22B weak scaling optimization

Qwen3-235B-A22B — Performance Optimization on Jupiter

Follows up on: Scaling on Jupiter task (link to previous task)
Config: megatron_moe_qwen235BA22B-sweep.yaml
W&B: OpenEuroLLM / qwen3-235B-A22B-scaling-jupiter

---

Goal

Close the gap between our current throughput and reference implementation numbers on 256 GPUs:

Reference	Precision	Target TFLOPs/s/GPU
Megatron-MoE-ModelZoo	BF16	276
Megatron Bridge / NeMo	FP8	238
Current (Jupiter, GH200 @ 680W)	FP8	170–175

Note: reference numbers are on non-power-capped hardware (H100). The ~60–100 TFLOPs gap is partially attributable to the 680W power cap on Jupiter GH200s (vs 900W rated), but there is likely additional headroom to recover through config tuning.

---

System

Jupiter — NVIDIA GH200 Grace Hopper Superchip, 96 GB HBM3, Quad-rail NDR200 InfiniBand
GH200 nodes are power-capped to 680W (rated 900W) — see #203

Software

deep_ep                          1.2.1+7febc6e
torch                            2.11.0a0+eb65b36914.nv26.2

Container: nvcr.io/nvidia/pytorch:26.03-py3 via Apptainer

---

Current parallelism config (baseline to beat)

From megatron_moe_qwen235BA22B.yaml

tensor_model_parallel_size: 2
pipeline_model_parallel_size: 8
expert_model_parallel_size: 16
expert_tensor_parallel_size: 1
num_layers_per_virtual_pipeline_stage: 4

global_batch_size: 2048 # For 64 nodes so 128 GAS
micro_batch_size: 1

 fp8: hybrid
 fp8_recipe: blockwise
 fp8_param_gather: true
 moe_router_dtype: "fp32"
 moe_router_force_load_balancing: true

moe_token_dispatcher_type: flex
moe_flex_dispatcher_backend: hybridep

recompute_granularity: selective
recompute_modules: [moe_act layernorm]
recompute_method: null
recompute_num_layers: null

---

Reference implementation configs (for comparison)

Megatron-MoE-ModelZoo H100 baseline (run_235b.sh) on BF16 with DeepEP:

TP=2  PP=8  VPP=4  EP=32  NNODES=32 (256 GPUs)  GBS=2048
flags:, --recompute-granularity selective, --recompute-modules moe_act layernorm, --moe-router-force-load-balancing

→276 TFLOPs/s/GPU
Megatron Bridge DGX-H100 (performance summary) on FP8:

256 GPUs  FP8-CS  GBS=2048  MBS=1  SeqLen=4096
TP=2  PP=8  VP=4  EP=32  GA=128

→ 233 TFLOPs/s/GPU