[feature request] INT4 inference support and flexgen like offloading.

Hello.

INT4 is accelerated on Ada Lovelace, Ampere and Turing GPU architectures tensor cores and can effectively halve VRAM requirements compared to INT8 (and that halves memory consumption in comparison to FP16), allowing for larger language models to easily run on consumer GPU hardware. According to the paper https://arxiv.org/abs/2212.09720 The accuracy loss is minimal.

There's also a new off-loading system called FlexGen which drastically improves speed compared to HF accelerate when offloading to the CPU. https://github.com/FMInference/FlexGen

I wish for Huggingface to implement these innovations into HF accelerate natively, so that LLMs can be run on consumer GPUs in an user-friendly, performant and intuitive way. Thank you.