[Feature] support llava1.5 w4a16 model? the model is so slower than origin fp16 model?

### Motivation

when i run this pipe = pipeline('liuhaotian/llava-v1.5-13b',
chat_template_config=ChatTemplateConfig(model_name='vicuna'), cache_max_entry_count=0.1)
some diffrent answeer given, but the origin model donnot occur,my promot is task+choices+format.
task = "your task is to find out what actions and events are included in the given image?"
choices = '''\nA. Someone are fighting\nB. Climbing the tree\nC. Climbing the wall
\nD. occupying roads to management and sell things\nE. Hanging clothes along the street\nF. Someone fell down, lay or sit on the ground
\nG. Climbing over a guardrail on the street\nH. Haphazard piles of materials
\nI. Talking on phone\nJ. Someone is smoking
\nK. None of the above'''
format = "\nAnswer with one or more option's letters from the given choices directly."
**is there some bug in this version or do i use the right way?at the same time the model is 3.0X slower than the origin fp16 model too?**

### Related resources

_No response_

### Additional context

_No response_