modelscope / modelscope/easydistill
白盒蒸馏是loss融合问题
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 475
- Forks
- 45
- PR merge metrics
- No merged PRs in 30d
Description
total_loss = (1 - self.kd_ratio) * lm_loss + self.kd_ratio * distil_loss
这里lm_loss和distil_loss在数量级上面差了近百倍，千倍，这样直接融合是否有意义？实际数据看lm_loss刚开始都是几十，最后收敛也到了0.1量级，但distil_loss是最开始也是0.001量级，收敛到0.0001量级，这样加权distill_loss基本没效果
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with easydistill/kd/train.py at line 131 and inspect how lm_loss, distil_loss, and kd_ratio are computed and combined. Compare their scales during training using the issue's reported values. Done should establish whether the distillation term contributes meaningfully and document or implement an agreed loss-balancing change, with validation from the training behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100