Request for Scripts to Merge QDoRA Adapters with Base Model for vLLM Inference

Hello,

I've successfully finetuned Llama-3 8B with QDoRA and am now looking to perform inference using vLLM. Could you provide guidance or scripts on how to merge the QDoRA adapters with the original base model? Additionally, does this process involve quantization and dequantization of the base model?

Thank you!

The issue names no files, tests, or entry points. Start by reviewing the QDoRA adapter output and the requirements for vLLM inference, including whether merging requires quantization or dequantization. Done would be a clear, agreed-upon script or documented procedure for preparing the fine-tuned model.

Written by the indexing model from the issue text.