QA-LoRA: Quantization Aware Low-Rank Adaptation

Nobody has claimed this yet.

Hi there 👋

Today I came across this paper: [QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models](https://arxiv.org/pdf/2309.14717.pdf).

So what's wrong with the current QLoRA?
QLoRA quantizes pretrained weights into `NF4`/`FP4` format, but keeps trainable LoRA weights (matrices A and B) in non-quantized form (`float16`/`bfloat16`). During each forward pass in each layer the pretrained weights are dequantized to the same dtype as LoRA weights. So that means that all calculations are still done in non-quantized form, which results in:
- No speed improvement compared to LoRA; actually it's slower as we have an overhead caused by quantization-dequantization process.
- If one wants to run a fine-tuned model on inference in a quantized form - the accuracy will drop.

That's what the authors of the paper are trying to solve: to train the model in full quantized form, so it is:
- Faster: no need to dequantize weights; weights are not in half-precision which should result in higher performance, since LLMs are IO-bound; in addition `INT4` dtype is used and "INT4 operators have been optimized by CUDA and are much faster in execution".
- No significant drop in accuracy on inference, especially if smaller n-bit quantization is used.

-----

Given that it's supposedly easy to implement plus brings that much improvement during fine-tuning and inference, I think that soon enough it will be all the rage and it's worth spending time on implementing.
But, of course, it's all up to @carmocca to decide.

In addition, I've noticed that for GPTQ quantization the [implementation repo](https://github.com/yuhuixu1993/qa-lora) uses [AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ). So it might be an answer to the [question](https://github.com/Lightning-AI/lit-gpt/issues/583) what implementation to use.

----

If it's decided to implement, I'd love to work on it: both AutoGPTQ and QA-LoRA.
But it will be after I finish the other work, which might not be soon 😞.

----

Links:
- Paper: https://arxiv.org/pdf/2309.14717.pdf
- Implementation: https://github.com/yuhuixu1993/qa-lora

Review the QA-LoRA paper and the linked qa-lora implementation, including its use of AutoGPTQ, then compare that approach with the project's existing quantization work. The issue names no files or tests, so implementation scope and completion criteria would need to be agreed before work begins.

Written by the indexing model from the issue text.