Fix Qwen3 MoE FLOP calculation
Open
bug
- Dominant language
- Python
- Stars
- 2k
- Forks
- 561
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 145
Description
from @guyueh1 on https://github.com/NVIDIA-NeMo/RL/pull/1034
> @ybgao-nvidia I wonder why we are putting Qwen3MoeConfig and Qwen3Config in the same if-branch? for moe shouldn't we use
>```
> moe_ffn_hidden_size=config.moe_intermediate_size,
> moe_router_topk=config.num_experts_per_tok,
>```
>instead of
>```
> moe_ffn_hidden_size=config.intermediate_size,
> moe_router_topk=1,
>```
>?
Contributor guide
Assessment
This issue has not been assessed yet.