NVIDIA/TensorRT-Edge-LLM

View on GitHub

High-performance, light-weight C++ LLM and VLM Inference Software for Physical AI

Stars: 563
Forks: 135
Open beginner issues: 0
Indexed issues: 71
Avg merge: 14h 13m
Merged PRs (30d): 1

Dominant language: Python
License: Apache-2.0
Last GitHub push: Sep 3, 2026
Latest indexed: Sep 19, 2026
Contributing guide: Contributing guide
Code of conduct: Code of conduct
Beginner labels: No beginner labels indexed

Beginner friendly Everything

Open All

71 open issues indexed

Decoding cannot be constrained: no per-step hook in the sampling path, and `logit_bias` cannot substitute for one Open

Difficulty 5/5 Over a week Newbie friendliness 38/100

NVIDIA/TensorRT-Edge-LLM#218 · Opened Sep 17, 2026
Direct export/build fails for Qwen3 VL FP8 fine-grained 2D block-FP8 checkpoints Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 55/100

NVIDIA/TensorRT-Edge-LLM#217 · Opened Sep 17, 2026
tensorrt-edgellm-build --dense nvfp4-qdq SIGSEGVs in a libnvinfer compiler worker thread on Jetson Thor (CC 11.0) for Cosmos-Reason2-2B; --dense fp16 builds fine from the same NVFP4 checkpoint Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA/TensorRT-Edge-LLM#216 · 2 comments · Opened Sep 17, 2026
llm_inference ignores temperature/top_p/top_k from the input JSON its own help tells you to set them in Open

Difficulty 2/5 1-3 hours Newbie friendliness 84/100

NVIDIA/TensorRT-Edge-LLM#211 · 1 comment · Opened Sep 12, 2026
Int4GroupwiseGemmPluginV2 fails silently when ENABLE_CUTE_DSL omits int4_fp16_gemm: "Failed to enqueue status -1" with no diagnostic Open

Difficulty 2/5 1-3 hours Newbie friendliness 35/100

NVIDIA/TensorRT-Edge-LLM#210 · Opened Sep 11, 2026
ONNX-less direct builder cannot resolve decoder tensors for multimodal cosmos3_edge, and renaming them yields a numerically wrong engine Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TensorRT-Edge-LLM#208 · Opened Sep 11, 2026
SM103 (B300 / Blackwell Ultra) is missing from five SM allowlists across the build and the attention path Open

Difficulty 4/5 3-5 days Newbie friendliness 58/100

NVIDIA/TensorRT-Edge-LLM#206 · Opened Sep 11, 2026
XQA NVRTC JIT passes no CUDA include path: cuda_fp16.h cannot find vector_types.h on x86 CUDA 12.x Open

Difficulty 3/5 1-2 days Newbie friendliness 68/100

NVIDIA/TensorRT-Edge-LLM#204 · Opened Sep 11, 2026
x86 CuteDSL cubins missing in the repo Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 55/100

NVIDIA/TensorRT-Edge-LLM#203 · Opened Sep 10, 2026
feat: Add base64 image support to llm_inference Open

Difficulty 3/5 1-2 days Newbie friendliness 67/100

NVIDIA/TensorRT-Edge-LLM#202 · Opened Sep 10, 2026
Experimental OpenAI-compatible server fails to start with: ModuleNotFoundError: No module named 'tensorrt' Open

Difficulty 4/5 3-5 days Newbie friendliness 55/100

NVIDIA/TensorRT-Edge-LLM#201 · 1 comment · Opened Sep 9, 2026
Qwen3-TTS CodePredictor (MTP) crashes with "No available kernel for the GQA" on Jetson AGX Orin Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA/TensorRT-Edge-LLM#198 · Opened Sep 4, 2026
Version mismatch while running engine files Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TensorRT-Edge-LLM#194 · 5 comments · Opened Aug 25, 2026
Qwen3.8-27B NVFP4 base engine build fails: Myelin removeEmptyProducersFromSubgraph not implemented for PLUGIN_V3 (Jetson Thor, TRT 10.13.3.9 and 10.14.1.48) Open

Difficulty 4/5 3-5 days Newbie friendliness 42/100

NVIDIA/TensorRT-Edge-LLM#192 · 4 comments · Opened Aug 24, 2026
feat: Add base64 image support to the OpenAI-compatible server Open

External PR Request

Difficulty 3/5 1-2 days Newbie friendliness 35/100

NVIDIA/TensorRT-Edge-LLM#191 · 1 comment · Opened Aug 24, 2026
feat: add InternVLA-N1-DualVLN vision-language navigation support Open

New Model

Difficulty 5/5 Over a week Newbie friendliness 25/100

NVIDIA/TensorRT-Edge-LLM#190 · Opened Aug 24, 2026
feat: quantized runtime input sidecars for Gemma 4 E-models Open

External PR Request

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA/TensorRT-Edge-LLM#188 · 1 comment · Opened Aug 22, 2026
[Jetson AGX Orin][v0.10.0] Native build may generate XQA kernels for SM80/86/89 instead of SM87 Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 68/100

NVIDIA/TensorRT-Edge-LLM#187 · 1 comment · Opened Aug 21, 2026
Support multimodal calibration when quantizing only the LLM backbone Open

Difficulty 4/5 3-5 days Newbie friendliness 54/100

NVIDIA/TensorRT-Edge-LLM#186 · 2 reactions · Opened Aug 20, 2026
The discrepancy between the cosine similarity of the intermediate vectors in the VIT engine of Qwen3-8B-VL and the PyTorch-based visual part leads to a decrease in accuracy. Open

bug

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA/TensorRT-Edge-LLM#185 · Opened Aug 19, 2026
fix: Jetson Orin + JetPack 6.2 (CUDA 12.6, listed as Compatible platform) fails to build with ENABLE_CUTE_DSL=ALL — missing cudaLibrary_t compat shim in cuteDslF16MoeRunner.h Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

NVIDIA/TensorRT-Edge-LLM#183 · 4 comments · 1 reaction · Opened Aug 17, 2026
handle_request() occasionally hangs forever (not a GPU fault) -- confirmed live with py-spy + gdb, no Xid errors Open

Difficulty 5/5 Over a week Newbie friendliness 28/100

NVIDIA/TensorRT-Edge-LLM#181 · 1 comment · Opened Aug 16, 2026
Idle scheduler loop retains last request's SamplingParams and re-validates it continuously (~30Hz), driving CPU + memory growth with zero active requests Open

Difficulty 4/5 3-5 days Newbie friendliness 55/100

NVIDIA/TensorRT-Edge-LLM#180 · 1 comment · Opened Aug 16, 2026
[Bug][Jetson AGX Orin] TensorRT slot.cpp decode assertion on every request after rebuilding at maxBatchSize=1 (fc_h_fusion workaround from #151) Open

Difficulty 5/5 Over a week Newbie friendliness 28/100

NVIDIA/TensorRT-Edge-LLM#179 · Opened Aug 15, 2026
[Bug] v0.10.0: prebuilt sm_87 (Jetson Orin) CuteDSL artifact missing fmha_v2_d* kernel modules, breaks all LLM engine builds Open

Difficulty 4/5 3-5 days Newbie friendliness 65/100

NVIDIA/TensorRT-Edge-LLM#178 · 3 comments · Opened Aug 15, 2026
[BUG] Qwen3-TTS CP-FP8 calibration failed, it mixes text and codec embedding domains Open

bug

Difficulty 5/5 Over a week Newbie friendliness 38/100

NVIDIA/TensorRT-Edge-LLM#175 · 1 comment · Opened Aug 13, 2026
CuTe DSL SSD head_dim=80 for Nemotron Mamba prefill on SM80+ (Blackwell D64 slab) Open

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA/TensorRT-Edge-LLM#174 · 4 comments · Opened Aug 13, 2026
[BUG] ONNX Export fails for Qwen3.5 MTP draft model with 4-bit quantized checkpoint due to unpacked mtp.fc.weight shape mismatch Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 55/100

NVIDIA/TensorRT-Edge-LLM#168 · 2 comments · Opened Aug 10, 2026
Exporting Alpamayo-R1-10B fails with AssertionError Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 55/100

NVIDIA/TensorRT-Edge-LLM#166 · 1 comment · Opened Aug 6, 2026
position_embeddings need check valid Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 78/100

NVIDIA/TensorRT-Edge-LLM#165 · 2 comments · Opened Aug 6, 2026
Bug: Garbage / repetitive output on Jetson AGX Thor (SM110) across all quantization formats using CuTe DSL FMHA Open

bug

Difficulty 5/5 Over a week Newbie friendliness 32/100

NVIDIA/TensorRT-Edge-LLM#164 · 2 comments · Opened Aug 5, 2026
Exporting Qwen2.5-VL AWQ checkpoints fails with KeyError: 'num_heads' Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 58/100

NVIDIA/TensorRT-Edge-LLM#163 · 1 comment · Opened Aug 5, 2026
[Bug] gen_cubins.py: XQA cubin generation writes to the module-level default path instead of --output_dir when workers do not inherit __main__ state Open

Difficulty 3/5 1-2 days Newbie friendliness 70/100

NVIDIA/TensorRT-Edge-LLM#161 · 3 comments · Opened Aug 4, 2026
Qwen3.5-35B-A3B GPTQ INT4 ONNX export ERROR using tensorrt edge llm 0.9.0 Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA/TensorRT-Edge-LLM#158 · 4 comments · Opened Jul 31, 2026
CUDA_ERROR_ILLEGAL_ADDRESS in XQA on Jetson Orin Nx for Gemma4 Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA/TensorRT-Edge-LLM#153 · 1 comment · Opened Jul 29, 2026
[Regression][Jetson AGX Thor] Layer-0 V tensor corruption in v0.9.1; v0.4.0 works correctly on the same JetPack 7.1 system Open

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA/TensorRT-Edge-LLM#151 · 6 comments · Opened Jul 28, 2026
LLM Quantization Cannot Load `text_dataset` (Default and Custom Values Fail) Open

bug

Difficulty 3/5 1-2 days Newbie friendliness 65/100

NVIDIA/TensorRT-Edge-LLM#150 · 2 comments · Opened Jul 27, 2026
[Bug][Export]: checkpoint loader can replace declared FP16/BF16 parameters with FP32 tensors Open

Difficulty 2/5 1-3 hours Newbie friendliness 72/100

NVIDIA/TensorRT-Edge-LLM#144 · 1 comment · Opened Jul 25, 2026
[Bug][Runtime]: context FMHA loads unrelated mask-type cubins and can fail with CUDA_ERROR_INVALID_IMAGE Open

Difficulty 4/5 3-5 days Newbie friendliness 55/100

NVIDIA/TensorRT-Edge-LLM#143 · Opened Jul 25, 2026
visual_quantization: invalid choice: 'mxfp8' Open

bug

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA/TensorRT-Edge-LLM#139 · 1 comment · Opened Jul 25, 2026
Support for Quantized Alpamayo Export Open

Difficulty 5/5 Over a week Newbie friendliness 30/100

NVIDIA/TensorRT-Edge-LLM#134 · 5 comments · Opened Jul 21, 2026
llm_bench profiling Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 48/100

NVIDIA/TensorRT-Edge-LLM#133 · 2 comments · Opened Jul 20, 2026
Alpamayo Support Open

New Model Roadmap

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA/TensorRT-Edge-LLM#130 · 3 comments · Opened Jul 8, 2026
[v0.9.0] Compile error: incorrect kernel module type names in cuteDslNvfp4MoeSm110Runner Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 76/100

NVIDIA/TensorRT-Edge-LLM#125 · 1 comment · Opened Jul 6, 2026
[Bug] Native Jetson Orin (sm_87) build: explicit -DCMAKE_CUDA_ARCHITECTURES ignored + CuTe DSL --wrap/cudart shim not propagated to static-lib consumers Open

bug

Difficulty 4/5 3-5 days Newbie friendliness 45/100

NVIDIA/TensorRT-Edge-LLM#117 · 2 comments · 1 reaction · Opened Jun 28, 2026
stbir_resize in resizeImage maybe calc wrong result Open

bug

Difficulty 2/5 1-3 hours Newbie friendliness 68/100

NVIDIA/TensorRT-Edge-LLM#116 · 4 comments · Opened Jun 26, 2026
Processing of frames that are already on device, in memory Open

New Feature Request

Difficulty 5/5 Over a week Newbie friendliness 35/100

NVIDIA/TensorRT-Edge-LLM#114 · 3 comments · Opened Jun 21, 2026
enable_thinking=True produces reasoning text without literal <think>...</think> tags — _ThinkingStateMachine cannot separate it from the final answer Open

Difficulty 4/5 3-5 days Newbie friendliness 45/100

NVIDIA/TensorRT-Edge-LLM#113 · 2 comments · Opened Jun 19, 2026
CustomVoice language conditioning bug confirmed in v0.8.0 — working fix applied (ref #87 / PR #98) Open

Difficulty 4/5 3-5 days Newbie friendliness 35/100

NVIDIA/TensorRT-Edge-LLM#109 · 2 comments · Opened Jun 17, 2026
Missing model_config argument in VLM visual ONNX export Open

Difficulty 2/5 1-3 hours Newbie friendliness 82/100

NVIDIA/TensorRT-Edge-LLM#107 · 2 comments · Opened Jun 12, 2026

NVIDIA/TensorRT-Edge-LLM

Get new issues in your inbox