[QUESTION] Why is expert_bias only supported with sigmoid gating and not softmax?

Hi Megatron-LM Team,

I have a quick question about the Mixture of Experts (MoE) implementation.

I've noticed that a learnable expert_bias term can be used with the sigmoid gating activation, but this option doesn't appear to be available when using softmax for the gating network.

Is this an intentional design choice? I'm curious if there's a specific theoretical or performance-related reason for this distinction.

Any insight you could provide would be greatly appreciated.

Thanks for your great work on this project!