NVIDIA-NeMo / NVIDIA-NeMo/Automodel

[KD][MoE] Add a validated MoE knowledge distillation recipe / example under examples/llm_kd/

Open
#3,299 0 comments 0 reactions 1 assignee View on GitHub

@akoumpa is already working on this.

Since Jul 30, 2026.

community-request enhancement waiting-on-maintainers
Dominant language
Python
Stars
963
Forks
318
Avg merge
3d 20h
Merged PRs (30d)
143

Description

Summary

MoE fine-tuning is well supported in AutoModel and validated at scale (GLM-5, MiniMax-M2.5, DeepSeek-V3.2, Step-3.5), but MoE knowledge distillation has no counterpart, there is no MoE example under examples/llm_kd/, and the KD recipe has diverged from train_ft.py in ways that break EP paths that already exist elsewhere in the codebase.

This request is for a validated, end-to-end MoE KD example covering the common combinations users are trying today. Even after the individual bugs are fixed, a working reference config is what turns "possible in principle" into "someone can copy-paste and run."

Ask
A recipe example (or set of examples) under examples/llm_kd/ covering at minimum:

  1. Dense student + MoE teacher using separate_meshes (once released). Reference pair: something small enough to fit on a single node Qwen3-4B student ← Qwen3-30B-A3B teacher is realistic on 8×H200.
  2. MoE ↔ MoE Reference pair: Qwen3-30B-A3B student ← larger MoE teacher, or smaller MoE ← Qwen3-30B-A3B teacher.
  3. Documented ep_size guidance for the student and teacher meshes, with a note on the ep_shard_size == 1 interaction with offload

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.