NVIDIA-NeMo / NVIDIA-NeMo/RL

Fix Qwen3 MoE FLOP calculation

Open
#1,144 0 comments 0 reactions 1 assignee Claimed by @ybgao-nvidia View on GitHub
bug
Dominant language
Python
Stars
2k
Forks
561
Avg merge
4d 5h
Merged PRs (30d)
145

Description

from @guyueh1 on https://github.com/NVIDIA-NeMo/RL/pull/1034

> @ybgao-nvidia I wonder why we are putting Qwen3MoeConfig and Qwen3Config in the same if-branch? for moe shouldn't we use
>```
> moe_ffn_hidden_size=config.moe_intermediate_size,
> moe_router_topk=config.num_experts_per_tok,
>```
>instead of
>```
> moe_ffn_hidden_size=config.intermediate_size,
> moe_router_topk=1,
>```
>?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.