Improvment to lora.py and flux lora.py

used this paper to implement the basic methodology into the lora.py network https://github.com/DAMO-NLP-SG/Inf-CLIP

network dim 32 sdxl now maintains a speed of 2.7 sec/it at a batch size of 40 for less than 24gb on a 4090. my flux implementation needs some help. i managed to get a batch size of 3 with no split on dim 32. using adafactor for both. please take a look let me know if i can help in any way.
[lora (2).txt](https://github.com/user-attachments/files/17532432/lora.2.txt)
[lora_flux.txt](https://github.com/user-attachments/files/17532433/lora_flux.txt)

im sure this can be more appropriately modified for flux, but i lack the expertise and familiarity of all your moving parts. let me know if theirs anything i can send over

Start by reviewing the Inf-CLIP paper linked in the issue and the attached lora (2).txt and lora_flux.txt files, then inspect the repository's lora.py and flux lora.py implementations. Compare the reported SDXL and Flux behavior with the existing methodology and benchmarks; done requires an agreed Flux-specific scope and validation criteria.

Written by the indexing model from the issue text.