[REQUEST] Usability of quantised models

I have used the compression tool of deepspeed to quantise a model. When I save the model state dict, the quantised tensors are indeed quantised, e.g. if I quantise using 8bit precision, I get that the tensor has at most 6084 unique values, corresponding to all the values that an 8bit register can take. Nonetheless, the precision of the corresponding tensors is fp16, and the actual values of the tensor are not integers.

It is not clear in any part of the documentation how should one:

1. Save a quantised model, where the dtypes of the quantised tensors are correct, e.g. int8.
2. Once one has a model with such dtypes, how to actually run the model for inference (https://www.deepspeed.ai/tutorials/inference-tutorial/#datatypes-and-quantized-models is not clear enough, as the first model object may use fp16 or fp32, which is already too big for edge devices).

Describe the solution you'd like
I would like a utility to save the quantised model with the correct data types, as well as a utility to load this model (with the internal knowledge of what the quantisation scales are).

Start by reviewing the DeepSpeed compression tool and the linked Inference tutorial, focusing on model state-dict saving, quantised dtypes, and inference datatypes. The requested outcome is a utility to save quantised tensors with appropriate dtypes and load them with their quantisation scales, but the issue does not identify files or tests.