Implement LoRA improvements

**Is your feature request related to a problem? Please describe.**
We'd like to incorporate the latest research on LoRA.

**Describe the solution you'd like**
Look at this paper and implement what makes sense: https://arxiv.org/abs/2410.09692

Start by reading the linked LoRA research paper and then locate the repository's existing LoRA implementation. Compare the current behavior with the paper's proposed improvements, but the issue does not name files, tests, or a specific scope, so the expected completed behavior needs clarification before implementation.

Written by the indexing model from the issue text.