Inferencing with adapter vs Inferencing with fused model

Hi everyone,

First of all let me say a big "THANK YOU" for your work !
I have successfully fine tuned phi-2. However I noticed the following:

• Inferencing with the fused model usually gives bad results. I use this code python -m mlx_lm.generate --model <fused-model-path> --prompt "hello". I don't see any effect of the fine tuning and sometimes it gives me nonsense.
• However when I use the following code, the results are much much better and I can see the fine tuning: python lora.py --model mlx_model --adapter-file adapters.npz --max-tokens 300 --prompt "<your-prompt-here>"

So I was wondering if someone can help me understand/explain this. My goal is to be able to use the model in a web app.

Thanks a lot !

Start by comparing the inference paths in mlx_lm.generate and lora.py, using the fused-model and adapter-file commands shown in the report. Reproduce the difference and trace where the fused model and adapter are loaded; done means documenting the cause and confirming the expected behavior for web-app inference.