deepseek-ai / deepseek-ai/FlashMLA
Why head_size must be 576 while q_head_dim of DeepSeek-V3 is only 192?
Open
- Dominant language
- C++
- Stars
- 12.9k
- Forks
- 1.2k
- Avg merge
- 4h 20m
- Merged PRs (30d)
- 2
Description
`head_size` here comes from `q.sizes()[3]`
But in 'modeling_deepseek.py' of DeepSeek-V3 model,
`q = q.view(bsz, q_len, self.num_heads, self.q_head_dim).transpose(1, 2)`
Here `self.q_head_dim = config.qk_nope_head_dim + config.qk_rope_head_dim` which is 128+64=192 according to 'config.json'.
How to understand this correctly?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.