Supporting distriminators like CLIP, CLIP Vision, DINO, SigLIP for semantic alignment

For semantic and visual alignment, adding other models for loss alignment can help retain and align models for faster training and better performance.

https://arxiv.org/abs/2410.06940 REPresentation Alignment (REPA) discusses some details of this for alignment. Additionally [they did a presentation on it](https://www.youtube.com/watch?v=hnu-mRLebhc).

Additionally other types of models like Aesthetic prediction models can be used for alignment.

To do this would require adding one or more types of models, caching the relevant embeddings, and then utilizing it in the loss calculations. The alignments might require more customized approach to listening and recording certain factors of the model.

This adds complexity that I want to be able to address but would like some feedback on some approaches.

One approach was related to [adding Redux support](https://github.com/kohya-ss/sd-scripts/pull/1838). I was able to add caching the SigLIP Vision embeddings and it uses a projection from SigLIP into T5-XXL sized embeddings. This process adds some complexity and it is scattered in various places.

Specifically the complexity is in dataset and caching where we add the embeddings but each implementation would need to add their own cached elements to the ImageInfo. In terms of caching it slots into the new Strategy system pretty well.

Last bit would be "hooking" into the model. With one example I have with DINO alignment, I'm doing a projection to DINOv2 features based on lower layers/blocks in the model, where the semantic details are most. And then training a separate AlignmentProjection module based on these layers and different timesteps to allow it to work with the random timesteps.

Downside with my current approach is the misalignment or poor alignment with the current model semantic alignment and the DINOv2 features. Training it to that goal rely's on the current model but could provide a future opportunity if it is built on relatively good models.

So if anyone has any feedback on these ideas, or their own ideas we can try to align them.

Start by reviewing the Redux support in pull request #1838, then trace the dataset and ImageInfo caching described in the issue alongside the Strategy system. Examine how a DINOv2 alignment projection would connect to model layers, timesteps, and loss calculations; the issue does not define a settled approach or concrete completion criteria.

Written by the indexing model from the issue text.