feat[5]: Hugging Face backbone adapter (prune, WLI bypass, distillation)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 0
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Goal
Iteration 5 of the SPADE implementation: make the framework directly applicable to HF-style residual Transformer backbones (GPT-2 pattern; same structure as CosyVoice 2 / LLaSA stacks). Provide layer-stack detection, state-dict-remapping layer pruning, leave-one-out block bypass for WLI, and a distillation trainer that reuses the existing SPADE losses via HF hidden states/attentions.
Acceptance
-
spade/adapters/hf.pydetects a model's layer ModuleList (transformer.h / model.layers / ...), prunes layers by state-dict remapping, and supports temporarily bypassing a block (WLI) -
compute_wliaccepts a pluggable bypass function so it works for any backbone -
HFDistillTrainerruns the SPADE composite loss on HF-style outputs (hidden_states/attentions/embedding) with dynamic layer matching - Tests with a locally-constructed tiny GPT-2 (no downloads): pruning correctness, WLI wiring, teacher frozen / student updated
Notes
Iteration 5 for: SPADE - Structured Pruning and Adaptive Distillation for Efficient LLM-TTS (arXiv:2509.20802)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the acceptance criteria and inspect the existing SPADE losses and compute_wli entry point before creating spade/adapters/hf.py. Define tests around a locally constructed tiny GPT-2: verify layer detection and pruning, WLI bypass wiring, and that the teacher stays frozen while the student updates. Done means HFDistillTrainer handles hidden_states, attentions, embedding, and dynamic layer matching.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- huggingface, python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100