NVIDIA / NVIDIA/TensorRT-Edge-LLM

Qwen3.8-27B NVFP4 base engine build fails: Myelin removeEmptyProducersFromSubgraph not implemented for PLUGIN_V3 (Jetson Thor, TRT 10.13.3.9 and 10.14.1.48)

Open
#192 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
563
Forks
135
Avg merge
14h 13m
Merged PRs (30d)
1

Description

Environment

  • Jetson AGX Thor (sm_110), L4T R38.4 (JetPack 7.1-class), CUDA 13.0
  • TensorRT-Edge-LLM 0.10.0 (bb29145)
  • TensorRT tested: 10.13.3.9-1+cuda13.0 (JetPack repo) AND 10.14.1.48-1+cuda13.0 (sbsa CUDA network repo) — same failure on both

Repro

Checkpoint: NVFP4 Qwen3.8-27B (ModelOpt NVFP4, group_size 16, MTP extra tensors), e.g. https://huggingface.co/lyf/Qwen3.8-27B-Heretic-ARA-NVFP4-MTP-VL

  1. tensorrt-edgellm-export CKPT OUT --mtp → succeeds (after locally patching two exporter issues: (a) system scipy 1.8 vs numpy 2.x — from numpy import Inf ImportError when the venv uses --system-site-packages; (b) GDN fusion concatenating single-element weight_scale_2/input_scale buffers into a 4-element vector, which TRT's ONNX parser rejects: importTRT_FP4DynamicQuantize: Scale input must be a scalar — fixed by keeping numel==1 buffers unconcatenated in fuse_gdn_input_projections).
  2. llm_build --onnxDir OUT/llm --engineDir ENG --maxBatchSize 1 --maxInputLen 4096 --maxKVCacheCapacity 32768 --maxVerifyTreeSize 4 --specBase
[ERROR] IBuilder::buildSerializedNetwork: Error Code 1: Internal Error
(Not implemented for node type PLUGIN_V3. In removeEmptyProducersFromSubgraph
at optimizer/myelin/rewrite/removeEmptyTensors.cpp:257)   // TRT 10.13
... removeEmptyTensors.cpp:306                              // TRT 10.14
  1. The MTP draft engine (--specDraft on OUT/mtp_draft) builds fine on both TRT versions. The draft ONNX contains only trt_edgellm::AttentionPlugin x1; the base ONNX additionally has trt_edgellm::gated_delta_net x48 and trt_edgellm::causal_conv1d x48, so the GDN plugins appear to be the PLUGIN_V3 nodes Myelin cannot handle.
  2. The experimental direct (ONNX-less) builder fails identically at the same Myelin pass, so this is not ONNX-path specific.

Questions

  • Which TRT version was Qwen3.8-27B Day-0 support validated against on Thor? Does it require JetPack 7.2 (CUDA 13.2 / TRT 10.16)?
  • Is there a builder flag/env to route GDN plugin nodes around the failing Myelin rewrite pass?

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the failure with tensorrt-edgellm-export and llm_build using the reported Qwen3.8-27B checkpoint and TensorRT versions. Compare the base and MTP draft ONNX graphs, especially the gated_delta_net and causal_conv1d PLUGIN_V3 nodes, then determine whether a supported TensorRT version or builder workaround avoids the Myelin rewrite failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, embedded-iot
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.