Error while loading custom finetuned QLoRA model in 4 bit : size mismatch for model.mm_projector.readout.0.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([8388608, 1]).

Hi Team,

I have successfully finetuned a QLoRA adapter on a custom dataset. When I try to load it in full precision, it gets loaded and works well

But this takes too much time and GPU memory to run inference. So I wanted to load the model in 4-bit precision. So I pass in the load_4bit parameter:

```python
model_path = '/home/apoorv/development/videollama2/VideoLLaMA2/work_dirs/videollama2_vllava/finetune_videollama2_vllava_qlora'
model_name = get_model_name_from_path(model_path)
tokenizer, model, processor, context_len = load_pretrained_model(model_path, None, model_name, load_4bit=True)
```

While running this I get the following error:

```
RuntimeError: Error(s) in loading state_dict for Videollama2MistralForCausalLM:
size mismatch for model.mm_projector.readout.0.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([8388608, 1]).
size mismatch for model.mm_projector.readout.2.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([8388608, 1]).
```

I have debugged and understood the reason that when we pass in the ```load_4bit=True```, the VideoLLama2 model params are loaded as 4bit params. The LLM part weights gets initialized from the base model (in this case mistralai/Mistral-7B-Instruct-v0.2) in 4-bit but the ```mm_projector``` does not (I am guessing it is initialized from the random values but in Params4bit class).

Line: https://github.com/DAMO-NLP-SG/VideoLLaMA2/blob/main/videollama2/model/builder.py#L76
```model = Videollama2MistralForCausalLM.from_pretrained(model_base, low_cpu_mem_usage=True, config=lora_cfg_pretrained, **kwargs)```

Now to initialize the weights for the ```mm_projector```, we try to load it from the previously saved ```non_lora_trainables.bin``` which was stored in full precision format.

Line: https://github.com/DAMO-NLP-SG/VideoLLaMA2/blob/main/videollama2/model/builder.py#L101
```model.load_state_dict(non_lora_trainables, strict=False)```

## Debugging Outputs

### Model params

```
**(Pdb) model.model.mm_projector.readout**
Sequential(
(0): Linear4bit(in_features=4096, out_features=4096, bias=True)
(1): GELU(approximate='none')
(2): Linear4bit(in_features=4096, out_features=4096, bias=True)
)
**(Pdb) model.model.mm_projector.readout[0]**
Linear4bit(in_features=4096, out_features=4096, bias=True)
**(Pdb) model.model.mm_projector.readout[0].weight**
Parameter containing:
Parameter(Params4bit([[193],
[108],
[250],
...,
[231],
[107],
[ 92]], device='cuda:1', dtype=torch.uint8))
**(Pdb) model.model.mm_projector.readout[0].weight.shape**
torch.Size([8388608, 1])
```

### Previously saved non_lora_trainables param

```
(Pdb) non_lora_trainables[ 'model.model.mm_projector.readout.0.weight'].shape
torch.Size([4096, 4096])
```

### Please advise on how to resolve this.

Start in videollama2/model/builder.py at lines 76 and 101, following load_pretrained_model with load_4bit=True and the subsequent non_lora_trainables load. Reproduce the custom QLoRA loading failure and compare the mm_projector parameter shapes; done means the adapter loads in 4-bit without the reported mismatch and supports inference.

Written by the indexing model from the issue text.