kohya-ss / kohya-ss/sd-scripts
Are there any plans to support DistillT5?
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
The project repo and whitepaper are [here](https://github.com/LifuWang-66/DistillT5). It's basically T5-XXL distilled to T5-Base size, and given the much smaller model size and VRAM requirements, could be very useful for training, especially for people with low VRAM that want or need to run training with the `train_t5xxl=true` flag.
Already tried to do my own naive patching by renaming the model layers to match the T5-XXL naming convention and by patching the `T5_CONFIG_JSON` in `library/flux_utils.py` to match DistillT5's config. That managed to get the model loaded, but training breaks on the first step because of architectural differences between T5-XXL and T5-Base that I'm not sure how to account for in the code...
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reviewing library/flux_utils.py, especially T5_CONFIG_JSON, and reproduce the first-step training failure with train_t5xxl=true using DistillT5. Done means DistillT5 loads and completes training beyond the first step while preserving support for the existing T5-XXL path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100