modelscope / modelscope/ms-swift
GKD蒸馏之后模型会发生灾难性遗忘，能否加上和reference model的正向KL散度
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 15.7k
- Forks
- 1.7k
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 136
Description
Describe the feature
Please describe the feature requested here(请在这里描述需求)
Paste any useful information
Paste any useful information, including papers, github links, etc.(请在这里描述其他有用的信息，比如相关的论文地址，github链接等)
Additional context
Add any other context or information here(其他信息可以写在这里)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
The issue requests adding a forward KL divergence against a reference model after GKD distillation, but names no files, tests, entry points, or acceptance criteria. Start by locating the GKD distillation implementation and its reference-model handling. Done should include an agreed design, implementation, and validation that addresses catastrophic forgetting.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100