NVIDIA / NVIDIA/TensorRT-Edge-LLM
Support for Quantized Alpamayo Export
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 563
- Forks
- 135
- Avg merge
- 14h 13m
- Merged PRs (30d)
- 1
Description
Detailed description of the requested feature
Alpamayo Example mentions that "only FP16 is supported for Alpamayo export in this release (v0.9.0)".
We would like to know if NVIDIA has plans to support quantized Alpamayo model export, such as FP8 or NVFP4, in the future.
We also noticed alpamayo-recipes repository provides a quantization script. We are wondering if we will be able to convert these quantized models using TensorRT Edge LLM for inference on edge devices.
Timeline
If there are plans to release this, any roadmap information would be very helpful.
Target hardware/use case
Hardware: DGX Spark (Also, is it possible to run TensorRT Edge LLM on DRIVE AGX Orin?)
CUDA: 13.0
Use case: Running quantized Alpamayo on edge devices.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked Alpamayo Example and the quantization script in alpamayo-recipes, then compare their FP16, FP8, and NVFP4 export paths with the TensorRT Edge LLM workflow. Done would require a confirmed implementation scope or roadmap for quantized Alpamayo export and inference, including whether DRIVE AGX Orin is supported.
Written by the indexing model from the issue text.
Assessment
- Domain
- machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100