ml-inory / ml-inory/SPADE

feat[5]: Hugging Face backbone adapter (prune, WLI bypass, distillation)

Open
#5 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
0
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Goal

Iteration 5 of the SPADE implementation: make the framework directly applicable to HF-style residual Transformer backbones (GPT-2 pattern; same structure as CosyVoice 2 / LLaSA stacks). Provide layer-stack detection, state-dict-remapping layer pruning, leave-one-out block bypass for WLI, and a distillation trainer that reuses the existing SPADE losses via HF hidden states/attentions.

Acceptance

  • spade/adapters/hf.py detects a model's layer ModuleList (transformer.h / model.layers / ...), prunes layers by state-dict remapping, and supports temporarily bypassing a block (WLI)
  • compute_wli accepts a pluggable bypass function so it works for any backbone
  • HFDistillTrainer runs the SPADE composite loss on HF-style outputs (hidden_states/attentions/embedding) with dynamic layer matching
  • Tests with a locally-constructed tiny GPT-2 (no downloads): pruning correctness, WLI wiring, teacher frozen / student updated

Notes

Iteration 5 for: SPADE - Structured Pruning and Adaptive Distillation for Efficient LLM-TTS (arXiv:2509.20802)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the acceptance criteria and inspect the existing SPADE losses and compute_wli entry point before creating spade/adapters/hf.py. Define tests around a locally constructed tiny GPT-2: verify layer detection and pruning, WLI bypass wiring, and that the teacher stays frozen while the student updates. Done means HFDistillTrainer handles hidden_states, attentions, embedding, and dynamic layer matching.

Written by the indexing model from the issue text.

Assessment

Tech stack
huggingface, python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.