kohya-ss / kohya-ss/sd-scripts

Are there any plans to support DistillT5?

Open
#2,052 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

The project repo and whitepaper are [here](https://github.com/LifuWang-66/DistillT5). It's basically T5-XXL distilled to T5-Base size, and given the much smaller model size and VRAM requirements, could be very useful for training, especially for people with low VRAM that want or need to run training with the `train_t5xxl=true` flag.

Already tried to do my own naive patching by renaming the model layers to match the T5-XXL naming convention and by patching the `T5_CONFIG_JSON` in `library/flux_utils.py` to match DistillT5's config. That managed to get the model loaded, but training breaks on the first step because of architectural differences between T5-XXL and T5-Base that I'm not sure how to account for in the code...

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing library/flux_utils.py, especially T5_CONFIG_JSON, and reproduce the first-step training failure with train_t5xxl=true using DistillT5. Done means DistillT5 loads and completes training beyond the first step while preserving support for the existing T5-XXL path.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.