deepjavalibrary / deepjavalibrary/djl
Support for FP8 quantization with TensorRT-LLM
- Dominant language
- Java
- Stars
- 4.9k
- Forks
- 760
- Avg merge
- 19h 26m
- Merged PRs (30d)
- 17
Description
DJL does not support (or has not documented support) for FP8 quantization ([docs](https://demodocs.djl.ai/docs/serving/serving/docs/lmi/user_guides/trt_llm_user_guide.html#quantization-support)).
FP8 is currently TensorRT-LLM's [recommended quantization technique](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/quantization-in-TRT-LLM.md#best-practices-to-choose-the-right-quantization-methods), with the lowest performance degradation with good speedup.
It would be great to support this in DJL. It should not affect any APIs other than adding options (I expect adding `option.quantization=fp8`).
Any users seeking a speedup or lower memory footprint would benefit from this change.
**Note**
This does contradict an [AWS blogpost](https://aws.amazon.com/blogs/machine-learning/boost-inference-performance-for-llms-with-new-amazon-sagemaker-containers/) but I expect this is an inaccuracy.
> as part of the latest LMI DLC release (0.25.0), enabling state-of-the-art optimizations like SmoothQuant, FP8, and continuous batching
Contributor guide
Research direction
Start with the linked TensorRT-LLM quantization-support documentation and the existing DJL serving configuration represented by option.quantization. Confirm the current supported values, then document or implement FP8 support and verify that the option enables TensorRT-LLM FP8 quantization without changing other APIs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100