[SDXL] Train SDXL refiner for subject generation

I've been using the scripts here to fine tune the base SDXL model for subject driven generation to good effect. However, I've found that adding the refiner step usually means that the refiner doesn't understand the subject, which often makes using the refiner worse with subject generation.

Currently the scripts do not support fine tuning the refiner to make generation better. When tuning the refiner, realistically we would only need to fine tune the text encoder, not the u-net. If this ability was added, I think that would be very beneficial.

No specific file or test is named. Start by tracing the SDXL training scripts and existing refiner handling, then determine how text-encoder-only fine-tuning could be supported; done means the scripts can train the refiner for subject generation without fine-tuning its U-Net.

Written by the indexing model from the issue text.