feat[3]: Adaptive multi-level distillation (Skew-KL losses, dynamic layer matching, trainer)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Goal
Iteration 3 of the SPADE implementation: the recovery stage. Implement the paper's composite distillation loss (Eq. 2): CE + Skew-KL logit alignment + latent/attention/embedding MSE, with dynamic layer matching so each student layer distills from the last teacher layer before the next retained layer.
Acceptance
-
spade/distill/losses.pyprovides CE, Skew KL (DistiLLM forward/reverse, masked), latent/attention/embedding MSE, and the SPADE composite loss (alpha=0.25 default) -
spade/distill/matching.pyprovides dynamic teacher-layer targets (last layer before next retained; final layer for the last student layer) and the static ablation -
spade/distill/trainer.pytrains the student against a frozen teacher with the composite loss and returns per-component metrics - Losses, matching, and trainer have passing pytest tests (including teacher frozen / student updated)
Notes
Iteration 3 for: SPADE - Structured Pruning and Adaptive Distillation for Efficient LLM-TTS (arXiv:2509.20802)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading Eq. 2 in the cited SPADE paper, then inspect the distillation work under spade/distill/. Implement and test losses.py, matching.py, and trainer.py against the listed acceptance criteria, including frozen-teacher and updated-student behavior. Run the pytest suite and confirm per-component metrics and both dynamic and static matching tests pass.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100