Alternating Captions

I originally opened this issue in the bmaltais repo, but this is probably the correct place: https://github.com/bmaltais/kohya_ss/issues/1836

Overtraining the text encoder is a real problem. If you look at online guides (such as those at civitai), they recommend using lower learning rates for the TE or not training it at all.

One really cool feature could be alternating captions for each image from a set. There should be an option to treat the captions files as a set of possible captions for the image (one possible caption per line), and a caption should randomly be selected every time an image is used for training.

For example, when training a character Lora, the caption file could look like:

a photo of name
a photo of name, a woman
name
this picture shows name
.....

Each time one of those captions would be randomly selected. This way we can get greater diversity in captions, which should make the TE training more robust.

I believe the Embeddings training script in Automatic1111 does something like this.

The issue describes selecting one caption at random from lines in each image's caption file during training, but names no files, tests, or entry points. Start by locating the training caption-loading path and clarify how caption sets should be recognized and sampled; done should mean repeated training uses can select different captions for the same image.

Written by the indexing model from the issue text.