NVIDIA / NVIDIA/TensorRT-Edge-LLM
[Bug] v0.10.0: prebuilt sm_87 (Jetson Orin) CuteDSL artifact missing fmha_v2_d* kernel modules, breaks all LLM engine builds
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 563
- Forks
- 135
- Avg merge
- 14h 13m
- Merged PRs (30d)
- 1
Description
Describe the bug
v0.10.0's CHANGELOG states: "Replaced the legacy embedded-cubin FMHA-v2 backend with CuTe DSL FMHA-v2 and removed the checked-in FMHA-v2 cubin artifacts." The new cpp/kernels/contextAttentionKernels/cuteDslFMHAV2Runner.h unconditionally declares LazyKernelModule<fmha_v2_d64_Kernel_Module_t> (and d128/d256/d512/paged/sw/bidirectional variants), but the prebuilt cpp/kernels/cuteDSLArtifact/aarch64/sm_87/ artifact checked into the repo at this same commit does not contain any fmha_v2_* headers (confirmed via find cpp/kernels/cuteDSLArtifact/aarch64/sm_87 -iname '*fmha*' → no results; only gemm/gdn/moe/int4_fp16_gemm/ffpa families are present). "fmha is always linked" per cpp/CMakeLists.txt comment, so this is not optional — it blocks compilation of edgellmCore entirely on Orin.
Since context-attention FMHA is required for essentially every LLM engine build, this appears to break llm_build for all models on native Jetson Orin (sm_87) at v0.10.0, not just spec-decode paths.
Steps to reproduce
# On Jetson Orin, JetPack 7.2, CUDA 13.2
git clone https://github.com/NVIDIA/TensorRT-Edge-LLM.git
cd TensorRT-Edge-LLM
git checkout v0.10.0
git submodule update --init --recursive
mkdir build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release -DTRT_PACKAGE_DIR=/usr \
-DCMAKE_TOOLCHAIN_FILE=cmake/aarch64_linux_toolchain.cmake \
-DEMBEDDED_TARGET=jetson-orin -DCUDA_CTK_VERSION=13.2 -DENABLE_CUTE_DSL=ALL
make -j$(nproc)
Actual behavior
cpp/kernels/contextAttentionKernels/cuteDslFMHAV2Runner.h:127:37: error: ‘fmha_v2_d64_Kernel_Module_t’ was not declared in this scope
127 | static detail::LazyKernelModule<fmha_v2_d64_Kernel_Module_t> sLLM_d64;
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated due to -Wfatal-errors.
make[2]: *** [cpp/CMakeFiles/edgellmCore.dir/build.make:216: cpp/CMakeFiles/edgellmCore.dir/kernels/contextAttentionKernels/cuteDslFMHAV2Runner.cpp.o] Error 1
Expected behavior
cpp/kernels/cuteDSLArtifact/aarch64/sm_87/ should ship the fmha_v2_d* CuteDSL kernel modules matching what the v0.10.0 header requires, same as (presumably) the x86 sm_110/sm_121 artifacts do.
System information (Edge Device)
- Platform: NVIDIA Jetson Orin NX 16GB (Seeed reComputer J4012)
- Software release: JetPack 7.2, CUDA 13.2
- CPU architecture: aarch64
- GPU compute capability: SM87
- Build type: Release
- TensorRT Edge-LLM version: v0.10.0 (
71dd1bae032e70771265917ec74d3ff4cad07a10) - CMake options:
-DEMBEDDED_TARGET=jetson-orin -DCUDA_CTK_VERSION=13.2 -DENABLE_CUTE_DSL=ALL(exact platform-recommended command from the installation docs)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with cpp/kernels/contextAttentionKernels/cuteDslFMHAV2Runner.h and inspect the aarch64/sm_87 contents under cpp/kernels/cuteDSLArtifact, then reproduce using the documented CMake command on Jetson Orin. Compare the declared fmha_v2 modules with the available artifacts; done means the required modules are present and the edgellmCore build completes for sm_87.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp
- Domain
- build-system, embedded-iot
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 65/100