deepjavalibrary / deepjavalibrary/djl

support dynamic quantization of pytorch

Open
#3,058 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
4.9k
Forks
759
Avg merge
16h 15m
Merged PRs (30d)
15

Description

## Description

dynamic quantization (weights quantized with activations read/stored in floating point and quantized for compute)

Will this change the current api?
- Model.quantize

Who will benefit from this enhancement?
Today, PyTorch supports the following backends for running quantized operators efficiently:

- x86 CPUs with AVX2 support or higher (without AVX2 some operations have inefficient implementations), via x86 optimized by [fbgemm](https://github.com/pytorch/FBGEMM) and [onednn](https://github.com/oneapi-src/oneDNN) (see the details at [RFC](https://github.com/pytorch/pytorch/issues/83888))

- ARM CPUs (typically found in mobile/embedded devices), via [qnnpack](https://github.com/pytorch/pytorch/tree/main/aten/src/ATen/native/quantized/cpu/qnnpack)

- (early prototype) support for NVidia GPU via [TensorRT](https://developer.nvidia.com/tensorrt) through fx2trt (to be open sourced)

## References

- [pytorch dynamic quantization](https://pytorch.org/tutorials/recipes/recipes/dynamic_quantization.html)
- [pytorch quantization](https://pytorch.org/docs/stable/quantization.html)
- [sentence-transformers quantization](https://www.sbert.net/examples/training/distillation/README.html#quantization)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.