deepseek-ai / deepseek-ai/profile-data
Is the backpropagation of weighted element-wise sum of MOE outputs missing in train.json ?
- Dominant language
- No language data
- Stars
- 1.2k
- Forks
- 152
- PR merge metrics
- No merged PRs in 30d
Description
I was trying to do some analysis with the train.json profiling and was unable to find the the backpropagation of weighted element-wise sum of moe outputs, which is the backpropagation of this calculation shown in the DeepSeek-V3 Technical Report:
My question is: is this neglected from the profiling on purpose or hidden somewhere in the deepEP operators?
My real motivation for this issue is to find out the data format for the MOE outputs, the activations saved for the gradient calculation of g_i,t shown in the equation above. Is it FP8, FP16/BF16, or even FP32?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.