[REQUEST] How to finetune ONLY certain subset of the network parameters

I have to add some LoRA layers by hand(without left) to a pre-trained Multi-modal model, to finetune the model for new data. I want Deepspeed to optimize ONLY the parameters from the LoRA layer rather than all the parameters. Like this

The platform is hugging face's transformers and Deepspeed.

Therefore I decorate the Trainer from HF's transformers, as below:

Unfortunately, it doesn't work, both LoRA and non-LoRa's weights are not changed during training. It seems that the optimizer in Deepspeed is not the same as that from Pytorch.

My question is, are there any ways that allow me to ONLY finetune certain subnet (LoRA) parameters with Deepspeed+Transformer's Trainer?

Start with the Hugging Face Transformers Trainer customization and DeepSpeed optimizer setup described in the issue, then trace how the LoRA and non-LoRA parameters reach the optimizer. There are no files or tests named; done means establishing whether only the selected LoRA parameters can be optimized and documenting or reproducing the result.