dtype: torch floating point value

I'm currently quantizing my model but to 4bit is there not way for me to say something like dtype=torch.float4 ? In the docs it states that there isn't but since transformers allows 4bit quantizing is it possible that I could do some minor changes to the code to make deepspeed work for 4 bit quantization? I keep getting the same error:

[2023-09-24 14:16:35,966] [WARNING] [config_utils.py:69:_process_deprecated_field] Config parameter mp_size is deprecated use 
tensor_parallel.tp_size instead
[2023-09-24 14:16:35,967] [INFO] [logging.py:96:log_dist] [Rank -1] quantize_bits = 8 mlp_extra_grouping = False, quantize_groups 
= 1
Traceback (most recent call last):
  File "/usr/local/testllama.py", line 484, in <module>
    ds_model = deepspeed.init_inference(
  File "/usr/local/lib/python3.10/dist-packages/deepspeed/__init__.py", line 342, in init_inference
    engine = InferenceEngine(model, config=ds_inference_config)
  File "/usr/local/lib/python3.10/dist-packages/deepspeed/inference/engine.py", line 110, in __init__
    self._convert_to_dtype(config)
  File "/usr/local/lib/python3.10/dist-packages/deepspeed/inference/engine.py", line 491, in _convert_to_dtype
    self.module.half()
  File "/usr/local/lib/python3.10/dist-packages/transformers/modeling_utils.py", line 2058, in half
    raise ValueError(
ValueError: `.half()` is not supported for quantized model. Please use the model as it is, since the model has already been casted to 
the correct `dtype`.

I'd also like to know where the numbers for the deepspeed inference times are coming from. According to some videos I watched deepspeed is capable of giving anywhere from a 2x to 12x speed boost. However, the only reliable looking numbers I've seen are from 20% to 3x. If I can get this working I'd be very happy with a 3x speed boost but still, I'd be incredibly happy with a 12x speed boost, just saying.

Also as a bit of an aside, unless I'm misunderstanding something, if ZERO-Infinity is capable of training models so enormous on limited resources, why isn't it used for inference?

Start with deepspeed/inference/engine.py at InferenceEngine._convert_to_dtype and compare the traceback with transformers/modeling_utils.py, where .half() rejects quantized models. Review config_utils.py for the quantization settings shown in the log. Done would require a clearly defined and validated approach for 4-bit quantized inference, but the issue also asks broader questions about speedups and ZERO-Infinity.