deepjavalibrary / deepjavalibrary/djl
support dynamic quantization of pytorch
- Dominant language
- Java
- Stars
- 4.9k
- Forks
- 759
- Avg merge
- 16h 15m
- Merged PRs (30d)
- 15
Description
## Description
dynamic quantization (weights quantized with activations read/stored in floating point and quantized for compute)
Will this change the current api?
- Model.quantize
Who will benefit from this enhancement?
Today, PyTorch supports the following backends for running quantized operators efficiently:
- x86 CPUs with AVX2 support or higher (without AVX2 some operations have inefficient implementations), via x86 optimized by [fbgemm](https://github.com/pytorch/FBGEMM) and [onednn](https://github.com/oneapi-src/oneDNN) (see the details at [RFC](https://github.com/pytorch/pytorch/issues/83888))
- ARM CPUs (typically found in mobile/embedded devices), via [qnnpack](https://github.com/pytorch/pytorch/tree/main/aten/src/ATen/native/quantized/cpu/qnnpack)
- (early prototype) support for NVidia GPU via [TensorRT](https://developer.nvidia.com/tensorrt) through fx2trt (to be open sourced)
## References
- [pytorch dynamic quantization](https://pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html)
- [pytorch quantization](https://pytorch.org/docs/stable/quantization.html)
- [sentence-transformers quantization](https://www.sbert.net/examples/training/distillation/README.html#quantization)
Contributor guide
Assessment
This issue has not been assessed yet.