NVIDIA / NVIDIA/TensorRT-Edge-LLM

[Bug] v0.10.0: prebuilt sm_87 (Jetson Orin) CuteDSL artifact missing fmha_v2_d* kernel modules, breaks all LLM engine builds

Open
#178 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
563
Forks
135
Avg merge
14h 13m
Merged PRs (30d)
1

Description

Describe the bug

v0.10.0's CHANGELOG states: "Replaced the legacy embedded-cubin FMHA-v2 backend with CuTe DSL FMHA-v2 and removed the checked-in FMHA-v2 cubin artifacts." The new cpp/kernels/contextAttentionKernels/cuteDslFMHAV2Runner.h unconditionally declares LazyKernelModule<fmha_v2_d64_Kernel_Module_t> (and d128/d256/d512/paged/sw/bidirectional variants), but the prebuilt cpp/kernels/cuteDSLArtifact/aarch64/sm_87/ artifact checked into the repo at this same commit does not contain any fmha_v2_* headers (confirmed via find cpp/kernels/cuteDSLArtifact/aarch64/sm_87 -iname '*fmha*' → no results; only gemm/gdn/moe/int4_fp16_gemm/ffpa families are present). "fmha is always linked" per cpp/CMakeLists.txt comment, so this is not optional — it blocks compilation of edgellmCore entirely on Orin.

Since context-attention FMHA is required for essentially every LLM engine build, this appears to break llm_build for all models on native Jetson Orin (sm_87) at v0.10.0, not just spec-decode paths.

Steps to reproduce
# On Jetson Orin, JetPack 7.2, CUDA 13.2
git clone https://github.com/NVIDIA/TensorRT-Edge-LLM.git
cd TensorRT-Edge-LLM
git checkout v0.10.0
git submodule update --init --recursive
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DTRT_PACKAGE_DIR=/usr \
  -DCMAKE_TOOLCHAIN_FILE=cmake/aarch64_linux_toolchain.cmake \
  -DEMBEDDED_TARGET=jetson-orin -DCUDA_CTK_VERSION=13.2 -DENABLE_CUTE_DSL=ALL
make -j$(nproc)
Actual behavior
cpp/kernels/contextAttentionKernels/cuteDslFMHAV2Runner.h:127:37: error: ‘fmha_v2_d64_Kernel_Module_t’ was not declared in this scope
  127 |     static detail::LazyKernelModule<fmha_v2_d64_Kernel_Module_t> sLLM_d64;
      |                                     ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated due to -Wfatal-errors.
make[2]: *** [cpp/CMakeFiles/edgellmCore.dir/build.make:216: cpp/CMakeFiles/edgellmCore.dir/kernels/contextAttentionKernels/cuteDslFMHAV2Runner.cpp.o] Error 1
Expected behavior

cpp/kernels/cuteDSLArtifact/aarch64/sm_87/ should ship the fmha_v2_d* CuteDSL kernel modules matching what the v0.10.0 header requires, same as (presumably) the x86 sm_110/sm_121 artifacts do.

System information (Edge Device)

  • Platform: NVIDIA Jetson Orin NX 16GB (Seeed reComputer J4012)
  • Software release: JetPack 7.2, CUDA 13.2
  • CPU architecture: aarch64
  • GPU compute capability: SM87
  • Build type: Release
  • TensorRT Edge-LLM version: v0.10.0 (71dd1bae032e70771265917ec74d3ff4cad07a10)
  • CMake options: -DEMBEDDED_TARGET=jetson-orin -DCUDA_CTK_VERSION=13.2 -DENABLE_CUTE_DSL=ALL (exact platform-recommended command from the installation docs)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with cpp/kernels/contextAttentionKernels/cuteDslFMHAV2Runner.h and inspect the aarch64/sm_87 contents under cpp/kernels/cuteDSLArtifact, then reproduce using the documented CMake command on Jetson Orin. Compare the declared fmha_v2 modules with the available artifacts; done means the required modules are present and the edgellmCore build completes for sm_87.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp
Domain
build-system, embedded-iot
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
65/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.