This Is The True Distillation
Open
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 47
- Forks
- 2
- PR merge metrics
- No merged PRs in 30d
Description
Rimo还是有点超脱于人类范畴了，看完这篇文章我已经震撼的语无伦次了，请让我组织以下语言：
- 权重有考虑开源吗
- 如果重新设计训练后模型的VAE，让Decoder支持4通道输入+16通道输出，会不会比直接强行分组4x4然后平均效果会好一点
- Clip这一块能跑通是不是因为直接冻结了OpenCLIP-ViT/bigG这一层clip。那这种clip不通用甚至没用clip的Qwen怎么用稍微阳间点的方式对齐clip，真的要手动加一层mlp处理映射吗🫠
- 这套方案能不能用于llm微调当中，通过直接提取隐层向量来训练（而不是和普通蒸馏一样在模型输出文本后再次训练），貌似还能减少显存和内存交换
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Begin by locating the repository's distillation implementation and checking how its VAE, OpenCLIP-ViT/bigG, Qwen alignment, and hidden-vector paths are represented. Done would require a maintainer-approved scope or design decision for these questions; the issue names no file, test, or concrete acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100