deepjavalibrary / deepjavalibrary/djl

Support for FP8 quantization with TensorRT-LLM

Open
#3,145 2 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
4.9k
Forks
760
Avg merge
19h 26m
Merged PRs (30d)
17

Description

DJL does not support (or has not documented support) for FP8 quantization ([docs](https://demodocs.djl.ai/docs/serving/serving/docs/lmi/user_guides/trt_llm_user_guide.html#quantization-support)).

FP8 is currently TensorRT-LLM's [recommended quantization technique](https://github.com/NVIDIA/TensorRT-LLM/blob/main/docs/source/blogs/quantization-in-TRT-LLM.md#best-practices-to-choose-the-right-quantization-methods), with the lowest performance degradation with good speedup.

It would be great to support this in DJL. It should not affect any APIs other than adding options (I expect adding `option.quantization=fp8`).

Any users seeking a speedup or lower memory footprint would benefit from this change.

**Note**
This does contradict an [AWS blogpost](https://aws.amazon.com/blogs/machine-learning/boost-inference-performance-for-llms-with-new-amazon-sagemaker-containers/) but I expect this is an inaccuracy.
> as part of the latest LMI DLC release (0.25.0), enabling state-of-the-art optimizations like SmoothQuant, FP8, and continuous batching

Contributor guide

Open the contributing guide

Research direction

Start with the linked TensorRT-LLM quantization-support documentation and the existing DJL serving configuration represented by option.quantization. Confirm the current supported values, then document or implement FP8 support and verify that the option enables TensorRT-LLM FP8 quantization without changing other APIs.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
machine-learning
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.