[Feature Request] MOBA：Mixture of Block Attention For Long-context
Open
community-request
enhancement
waiting-on-customer
- Dominant language
- Python
- Stars
- 17.9k
- Forks
- 4.5k
- Avg merge
- 4d 6h
- Merged PRs (30d)
- 271
Description
**Summary**
Request implementation of MoBA Attention, the technique used by Kimi for efficient long-context training and inference.
**Current state**
Kimi official has already placed the basic code in their repository. Are there any plans to integrate it into megatron-lm？
MoBA：https://github.com/MoonshotAI/MoBA
Contributor guide
Assessment
This issue has not been assessed yet.