NVIDIA-NeMo / NVIDIA-NeMo/Automodel
[KD][MoE] Add a validated MoE knowledge distillation recipe / example under examples/llm_kd/
@akoumpa is already working on this.
Since Jul 30, 2026.
- Dominant language
- Python
- Stars
- 963
- Forks
- 318
- Avg merge
- 3d 20h
- Merged PRs (30d)
- 143
Description
Summary
MoE fine-tuning is well supported in AutoModel and validated at scale (GLM-5, MiniMax-M2.5, DeepSeek-V3.2, Step-3.5), but MoE knowledge distillation has no counterpart, there is no MoE example under examples/llm_kd/, and the KD recipe has diverged from train_ft.py in ways that break EP paths that already exist elsewhere in the codebase.
This request is for a validated, end-to-end MoE KD example covering the common combinations users are trying today. Even after the individual bugs are fixed, a working reference config is what turns "possible in principle" into "someone can copy-paste and run."
Ask
A recipe example (or set of examples) under examples/llm_kd/ covering at minimum:
- Dense student + MoE teacher using separate_meshes (once released). Reference pair: something small enough to fit on a single node Qwen3-4B student ← Qwen3-30B-A3B teacher is realistic on 8×H200.
- MoE ↔ MoE Reference pair:
Qwen3-30B-A3Bstudent ← larger MoE teacher, or smaller MoE ←Qwen3-30B-A3Bteacher. - Documented ep_size guidance for the student and teacher meshes, with a note on the ep_shard_size == 1 interaction with offload
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.