kohya-ss / kohya-ss/sd-scripts
Best Code for Full SDXL finetuning?
- Dominant language
- Python
- Stars
- 7.2k
- Forks
- 1.2k
- Avg merge
- 11m
- Merged PRs (30d)
- 2
Description
I am currently working on full SDXL fine-tuning but have encountered challenges in finding the best code due to conflicting information from various sources. Previously, I used the GitHub repository [kohya-trainer](https://github.com/qaneel/kohya-trainer), which was functional for many cases. However, it has a limitation: as I learned from running the SDXL pipeline in ComfyUI, there are two text encoders for G and L. While I may be mistaken, ComfyUI suggests this. I also observed that the SDXL 1.0 files on Hugging Face support two prompts.
The kohya-trainer code only allows for a single prompt as a tag or caption, which poses a challenge for my use case. To address this, I found another full fine-tuning code in the [sd-scripts repository](https://github.com/mio-nyan/sd-scripts/tree/dev) that accommodates two different prompts. However, I'm uncertain whether to trust this code.
To illustrate the differences, here are examples of the metadata files from both repositories:
1. **https://github.com/qaneel/kohya-trainer**
{
"filename": {
"tags": "tag for text encoder",
"train_resolution": [
896,
1152}
]
},
2. **https://github.com/mio-nyan/sd-scripts/tree/dev**
{
"filename": {
"captionG": "caption for (G) text_encoder2",
"captionL": "caption for (L) text_encoder1",
"train_resolution": [
896,
1152
]
},
}
So based on this can anyone suggest me what or which gitub repo to choose for full finetuning of SDXL using both the text encoder..
Second problem which I see in multiple places was related to the token size of clip models as we know the base clip has the token size of 77 but after the release of the LongCLIP https://github.com/beichenzbc/Long-CLIP/tree/main they were able to increase the token length of clip from 77 to 248 the reason which they gave for increasing the token size of the clip was that that they mentioned that the actually effective length of clip is just 20 tokens only which is very less than 77 itself, but I still didn't found any code of finetuning of sdxl using this LongClip model, I hope that this SDXL community of typical generative AI people can help me in solving this issue,
Finally if anyone want to use there fine-tuned SDXL model with ComfyUI they can look at this github repo of SeaArtLab **https://github.com/SeaArtLab/ComfyUI-Long-CLIP** and still If you are not able to run longclip with the comfyui or any sdxl pipeline they can ask me for sure.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by comparing the SDXL fine-tuning behavior described for kohya-trainer and the sd-scripts dev branch, focusing on the tags versus captionG/captionL metadata. Then review the linked Long-CLIP and ComfyUI-Long-CLIP repositories to determine whether their 248-token setup is supported. Done means identifying a validated full-finetuning path using both text encoders and documenting Long-CLIP compatibility.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 18/100