[BUG] 8 bit quantized inference not as fast as hoped for?

Describe the bug
I am using the 4 bit post init quantization approach. I was hopping it would make inference faster in addition to saving memory.
But it is not the case.

To Reproduce
Quantize a model such as StarCoder.

Expected behavior
inference being memory bound due to reading model weights, I thought it would be almost linearly faster when quantizing.
But it is instead 2x slower.
Is that because the weights are first dequantized, then torch linear is used? I guess it would be faster if it was fused together?

ds_report output
Please run ds_report to give us details about your setup.

Screenshots
If applicable, add screenshots to help explain your problem.

System info (please complete the following information):

• OS: [e.g. Ubuntu 18.04]
• GPU count and types [e.g. two machines with x8 A100s each]
• (if applicable) what DeepSpeed-MII version are you using
• (if applicable) Hugging Face Transformers/Accelerate/etc. versions
• Python version
• Any other relevant info about your setup

Docker context
Are you using a specific docker image that you can share?

Additional context
Add any other context about the problem here.

Start by running ds_report and collecting the missing OS, GPU, DeepSpeed-MII, Transformers, Accelerate, and Python versions. Reproduce the StarCoder 4-bit post-init quantization case and compare inference speed and memory use with the unquantized model; done means the slowdown is explained or a specific quantization performance issue is isolated.