Training the SDXL text encoder with sdxl_train.py adds a pink / purple color to output images

Hi there,

I'm using the recently fixed SDXL text encoder training support to train the text encoder with some new terms. This works, and I can successfully use the trained terms to generate matching images.

However, I am finding that whenever I train the text encoder, my generated images (in A1111) have a very strong pink / purple color across all of the image. If I disable text encoder training, and only train the UNet, I do not see the pink / purple tint.

The issue is especially noticeable if I _only_ train the text encoder, and not the UNet, with sdxl_train.py. If I also train the UNet, it looks as though the image learning from the UNet "undoes" some of the purple effect. However, it still does not go away completely.

What kind of issue might cause this color tinting, and is there any way to work around it?

Reproduce the issue with sdxl_train.py by comparing text-encoder-only training with UNet-only training, then inspect the text-encoder training path for the source of the tint. Use A1111 to compare generated outputs; done means identifying the cause and verifying a workaround that removes the pink/purple cast without disabling the intended training.

Written by the indexing model from the issue text.