NVIDIA / NVIDIA/TensorRT-Edge-LLM
fix: Jetson Orin + JetPack 6.2 (CUDA 12.6, listed as Compatible platform) fails to build with ENABLE_CUTE_DSL=ALL — missing cudaLibrary_t compat shim in cuteDslF16MoeRunner.h
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 563
- Forks
- 135
- Avg merge
- 14h 13m
- Merged PRs (30d)
- 1
Description
Describe the bug
The documented build flow for Jetson Orin + JetPack 6.2+ — a platform listed
in the Official Support Matrix
as Compatible ("expected to work with the stated constraints") — fails to
compile in 0.10.0:
In file included from cpp/kernels/cuteDSLArtifact/aarch64/sm_87/include/cutedsl_f16_moe_all.h:3,
from cpp/kernels/moe/f16_cutedsl/cuteDslF16MoeRunner.h:28,
from cpp/kernels/moe/f16_cutedsl/cuteDslF16MoeRunner.cpp:20:
cpp/kernels/cuteDSLArtifact/aarch64/sm_87/include/f16_moe_ampere_grouped_fp16.h:21:5:
error: 'cudaLibrary_t' does not name a type; did you mean 'cudaArray_t'?
21 | cudaLibrary_t module;
Impact: blocker — the support matrix promises Jetson Orin / JetPack 6.2+ /
CUDA 12.6 works, and the Installation guide gives a dedicated "JetPack 6.2+
Orin" CMake command that includes -DENABLE_CUTE_DSL=ALL ("All commands
enable CuTe DSL kernels because Qwen3.5 and several other model paths require
them"). Following that documented command does not build.
Root cause: cudaLibrary_t is a CUDA runtime type introduced in CUDA
12.8, but AOT-generated CuTe DSL artifact headers use it unconditionally. All
other CuTe DSL runner headers compensate with a compat typedef
(typedef CUlibrary cudaLibrary_t + inline cudaLibraryUnload) guarded by
TRT_EDGELLM_CUDA_LIBRARY_T_COMPAT — which the root CMakeLists.txt
auto-defines for EMBEDDED_TARGET=jetson-orin (see comment at
CMakeLists.txt:92-96). cuteDslF16MoeRunner.h is the only runner header
missing this block:
| Header | Compat block |
|---|---|
cpp/kernels/gdnKernels/cuteDslGDNRunner.h |
✅ |
cpp/kernels/mamba/cuteDslSSDRunner.h |
✅ |
cpp/kernels/moe/nvfp4_cutedsl/cuteDslNvfp4MoeRunner.h |
✅ |
cpp/kernels/talkerMLPKernels/cuteDslGemmRunner.h |
✅ |
cpp/plugins/int4GroupwiseGemmPluginV2/cuteDslInt4Gemm.cpp / cuteDslInt4Gemv.cpp |
✅ |
cpp/kernels/moe/f16_cutedsl/cuteDslF16MoeRunner.h |
❌ missing |
Related gap: kernelSrcs/cuteDSLPrebuilt/ only ships cuda13 tarballs
(e.g. cutedsl_aarch64_sm_87_cuda13.tar.gz), so CUDA 12.6 Orin users must
generate artifacts locally via build_cutedsl.py — the failure reproduces
with a freshly generated cuda12/sm_87 artifact, so it is independent of
artifact provenance.
Proposed fix: add the identical compat block to cuteDslF16MoeRunner.h.
I have a verified patch (full build succeeds, binaries produced) and can send
a PR if approved.
Steps/Code to reproduce bug
- Generate a local CUDA 12 / sm_87 CuTe DSL artifact (no cuda12 prebuilt
tarball ships in the repo):
python kernelSrcs/build_cutedsl.py --kernels ALL --gpu_arch sm_87 --cuda-version 12
- Configure and build with the exact "JetPack 6.2+ Orin" command from the
Installation guide:
Build configuration:
mkdir -p build && cd build
cmake .. \
-DCMAKE_BUILD_TYPE=Release \
-DTRT_PACKAGE_DIR=/usr \
-DCMAKE_TOOLCHAIN_FILE=cmake/aarch64_linux_toolchain.cmake \
-DEMBEDDED_TARGET=jetson-orin \
-DCUDA_CTK_VERSION=12.6 \
-DENABLE_CUTE_DSL=ALL
make -j$(nproc)
Runtime command used:
# N/A — compile-time failure; no runtime reached.
Expected behavior
The build succeeds as promised by the support matrix for the Compatible
Jetson Orin / JetPack 6.2+ / CUDA 12.6 combination. Verified locally: adding
the same compat block used by the sibling runner headers fixes the build:
#if defined(CUTE_DSL_F16_MOE_ENABLED)
+#include <cuda.h>
+#if defined(TRT_EDGELLM_CUDA_LIBRARY_T_COMPAT)
+#include <cuda_runtime.h>
+#if CUDA_VERSION >= 12000 && CUDA_VERSION < 12080
+typedef CUlibrary cudaLibrary_t;
+static inline cudaError_t cudaLibraryUnload(cudaLibrary_t lib)
+{
+ CUresult r = cuLibraryUnload(lib);
+ return static_cast<cudaError_t>(r);
+}
+#endif // CUDA_VERSION >= 12000 && CUDA_VERSION < 12080
+#endif // TRT_EDGELLM_CUDA_LIBRARY_T_COMPAT
+
#include "kernels/cuteDslModuleLoader.h"
After the patch, make -j reaches 100% and produces llm_build /
llm_inference / visual_build.
System information (Edge Device)
- Platform: NVIDIA Jetson AGX Orin Developer Kit
- Software release: JetPack 6.2
- CPU architecture: aarch64
- GPU compute capability: SM87
- Total device memory: 61Gi (unified)
- Build type: Release
- Library versions:
- TensorRT Edge-LLM version or commit hash: 0.10.0 (release commit
0ce6ce2) - CUDA: 12.6 (V12.6.68)
- TensorRT: 10.3.0.30
- C++ compiler: GCC 11.4.0
- TensorRT Edge-LLM version or commit hash: 0.10.0 (release commit
- CMake options used:
- CMAKE_TOOLCHAIN_FILE:
cmake/aarch64_linux_toolchain.cmake - EMBEDDED_TARGET:
jetson-orin - TRT_PACKAGE_DIR:
/usr
- CMAKE_TOOLCHAIN_FILE:
- Any other details that may help:
ENABLE_CUTE_DSL=ALL,CUDA_CTK_VERSION=12.6- CuTe DSL artifact metadata groups:
f16_moe, fmha, gdn, gemm, int4_fp16_gemm, ssd(generated withnvidia-cutlass-dsl[cu12]==4.6.1) - Failure is specific to the
f16_moegroup; builds without it are
unaffected - The same failure applies to x86 CUDA 12 hosts passing
-DCMAKE_CXX_FLAGS=-DTRT_EDGELLM_CUDA_LIBRARY_T_COMPATper
kernelSrcs/int4_fp16_gemm_cutedsl/README.md
`
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in cpp/kernels/moe/f16_cutedsl/cuteDslF16MoeRunner.h and compare its CUDA compatibility handling with the sibling CuTe DSL runner headers named in the issue. Reproduce with kernelSrcs/build_cutedsl.py and the documented Jetson Orin CMake command; done means the CUDA 12.6 build reaches 100% and produces the listed binaries.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp, python
- Domain
- build-system, embedded-iot
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 78/100