kohya-ss / kohya-ss/sd-scripts

weighted captions for SDXL

Open
#1,437 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
7.2k
Forks
1.2k
Avg merge
11m
Merged PRs (30d)
2

Description

Hi, I really, really need weighted captions and I see this in sdxl_train_network:
```
# Get the text embedding for conditioning
# TODO support weighted captions
# if args.weighted_captions:
# encoder_hidden_states = get_weighted_text_embeddings(
# tokenizer,
# text_encoder,
# batch["captions"],
# accelerator.device,
# args.max_token_length // 75 if args.max_token_length else 1,
# clip_skip=args.clip_skip,
# )
# else:
```
I've looked a little around in the scripts, looked at train_util.get_hidden_states_sdxl, sdxl_gen_img and library/custom_train_functions but I don't have any experience in this so I don't know how to implement it myself, does anyone have any suggestions how to get this working quick and dirty? there are a couple of different functions with similiar functionality already implemented but I know too little to meaningfully copy-paste them...

Contributor guide

No contributing guide indexed for this repository

Research direction

Start at the weighted-captions TODO in sdxl_train_network and compare the existing related paths in train_util.get_hidden_states_sdxl, sdxl_gen_img, and library/custom_train_functions. Trace how captions become text embeddings in these paths. Done means SDXL training accepts and applies weighted captions through the intended conditioning flow.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.