DPO alignment doesn't work on Lora models as suggested

You claim that "[In practice, we find comparable performance for both full and LoRA fine-tuning, with the latter having the advantage of producing small adapter weights that are fast to upload and download from the Hugging Face Hub.](https://github.com/huggingface/alignment-handbook/tree/main/scripts#:~:text=In%20practice%2C%20we%20find%20comparable%20performance%20for%20both%20full%20and%20LoRA%20fine%2Dtuning%2C%20with%20the%20latter%20having%20the%20advantage%20of%20producing%20small%20adapter%20weights%20that%20are%20fast%20to%20upload%20and%20download%20from%20the%20Hugging%20Face%20Hub.)"

However, when I try the Lora model DPO-aligned LLM that you have trained, [alignment-handbook/zephyr-7b-dpo-lora](https://huggingface.co/alignment-handbook/zephyr-7b-dpo-lora), I experience a total performance degradation.
Here is an example of model output that seems confused:

Even the training loss indicates that the model has not learned much

Here is the training loss for the full model DPO alignment.

Would you please do a clarification? Is my observation different from what you have experienced?

Thanks