NVIDIA-NeMo / NVIDIA-NeMo/Automodel
[Tracking] Kimi K3 roadmap
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 963
- Forks
- 318
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 143
Description
Kimi K3 training currently runs only on GB200. This roadmap tracks GB200 support and optimization.
Last refreshed: 2026-08-19.
Status: ✅ validated · 🚧 waiting on validation · 📋 planned · ❌ known broken
1. Core Training
| Feature | GB200 | Notes |
|---|---|---|
| Basic training (EP + PP + FSDP) | ✅ | Implemented in #3259; GB200 recipe and CI topology corrected in #3420 |
| Vision training | 🚧 | Implementation landed in #3259; full-model recipe validation pending |
2. Fusion & Performance
| Feature | GB200 | Notes |
|---|---|---|
| Transformer Engine Gated MLA | ✅ | Implemented in #3259; correctness validated without CP, full-model performance benchmark planned |
| FlashKDA | 📋 | P2; implementation PR pending |
| MoonEP | 📋 | P0; tracked in #3251, implementation PR pending |
| Attention Residuals fusion | 📋 | P2; implementation PR pending |
| Grouped GEMM + SiTU-GLU | 📋 | P2; base grouped-expert support landed in #3259, fused path PR pending |
3. Parallelism
| Feature | GB200 | Notes |
|---|---|---|
| PP + EP + FSDP | ✅ | Validated in #3259; GB200 recipe follow-up in #3420 |
| Context Parallel | 🚧 | CP1/CP2 parity validated in #3259; CP recipe validation pending |
| HybridEP | ✅ | Validated in #3259 |
4. Related Kimi K3 Work
| Feature | Status | PR |
|---|---|---|
| DSpark draft training | ✅ | #3263 |
| EAGLE-3 draft training | ✅ | #3286 |
| DFlash draft training | 🚧 | #3287 |
| Expert LoRA HF export | ✅ | #3435 |
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the status table and the linked PRs and issue, especially #3251, #3259, #3420, and #3435. This is a broad roadmap rather than a self-contained task; completion would require implementing or validating the remaining GB200 items and updating their statuses.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python, pytorch
- Domain
- distributed-systems, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100