Unable to load *ANY BASE MODEL* in 4bit

Hi VideoLLaMA Team,

I am facing issues while loading all the base models in 4-bit precision. The following lines try to load the ```mm_projector_weights``` which are stored in 16-bit precision into a model that requires the weights in 4bit leading to errors:

*Code used for loading the models for inference*

```python
model_path = 'DAMO-NLP-SG/VideoLLaMA2-8x7B-Base'
model, processor, tokenizer = model_init(model_path, load_4bit=True)
```

*Problematic part of the Code:*
Lines: https://github.com/DAMO-NLP-SG/VideoLLaMA2/blob/main/videollama2/model/__init__.py#L171-L172
```python
mm_projector_weights = load_mm_projector(model_path, token=token)
model.load_state_dict(mm_projector_weights, strict=False)
```

*Error:*
```
RuntimeError: Error(s) in loading state_dict for Videollama2MistralForCausalLM:
size mismatch for model.mm_projector.readout.0.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([8388608, 1]).
size mismatch for model.mm_projector.readout.2.weight: copying a param with shape torch.Size([4096, 4096]) from checkpoint, the shape in current model is torch.Size([8388608, 1]).
```

**How can we use the 16-bit stored weights of the ```mm_projector_weights``` in 4-bit models?**