Integrate Intel Neural Compressor tool to quantize fp32 model

🚀 Feature

Intel Neural Compressor tool is an open-source Python library running on Intel CPUs and GPUs, which delivers unified interfaces across multiple deep learning frameworks for popular network compression technologies, such as quantization, pruning, knowledge distillation.

Motivation

from discussion in https://github.com/PyTorchLightning/pytorch-lightning/pull/10793
Neural Compressor supports automatic accuracy-driven tuning strategies to help users quickly find out the best-quantized model, and it supports three modes: post-training static quantization, post-training dynamic quantization and quantization aware training. It also implements different weight pruning algorithms to generate pruned models with predefined sparsity goals and supports knowledge distillation to distill the knowledge from the teacher model to the student model.

Pitch

Alternatives

Additional context

Now INC only supports the LightningModules which has a backbone model like https://github.com/PenghuiCheng/pytorch-lightning/blob/e5a9d6fb38db5c835672e45c6fcdc21444e2d7e1/pl_examples/quantization/imagenet.py#L87.

The test result on AWS c6i.16xlarge instance [Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz]
BS=1 Cores/instance=4 Instance=8

cc @borda