Question: Qwen3 native KV-cache decode engine fails to build on SM120/SM121 (DGX Spark / GB10)

Topic

Installation or source build

Question

Summary

On a DGX Spark (GB10, compute capability 12.1), trtmc build succeeds for Qwen2.5
models but fails for every Qwen3 model. The prefill engine always builds; the
decode engine fails inside TensorRT's Myelin attention fusion.

The failure is confined to the Qwen3 native KV-cache decode path
(IAttention + key_value_lengths + IKVCacheUpdateLayer). A correct
workaround exists (--dynamic-kv-cache, see below), so this is not a hard
blocker — but the current failure mode is an opaque internal TensorRT exception
rather than an actionable message, and the SM120/121 limitation is undocumented.

Environment

GPU	NVIDIA GB10 (DGX Spark), compute capability 12.1 (sm_121)
Driver	580.173.02
Arch / OS	aarch64, Ubuntu 24.04.4 LTS
CUDA	13.0 (V13.0.88)
TensorRT	11.1.0.106 (libnvinfer-dev 11.1.0.106-1+cuda13.3; pip tensorrt==11.1.0.106)
Build	host build (no container), -DCMAKE_CUDA_ARCHITECTURES=121-real, TRT backend ABI 11_1

The C++ build itself is fine: trtmc, libtrtmc_backend_trt_11_1.so, and
libtrtmc_model_qwen.so all compile, and Qwen2.5 bundles run correctly.

Reproduction

trtmc build Qwen/Qwen3-0.6B -o qwen3-0.6b.bundle

[trtmc build] Prefill engine built [22.2s] (1144.9 MB)
Internal Error: MyelinCheckException: decompose_composite_ops.cpp:29:
  CHECK(composite_op->is_decomposable()) failed.
  Attention operation was not supported by a dedicated kernel.
[TRT] [E] Error Code: 10: Internal Error (Could not find any implementation for
  node {ForeignNode[(Unnamed Layer* 2) [Cast]...(Unnamed Layer* 2837) [Cast]]})
Error: dual-profile decoder engine build failed

What we isolated

Every row below was run on this machine.

Model	arch	head_dim	heads x head_dim vs hidden	QK-norm	Result
Qwen2.5-0.5B-Instruct	qwen2	64	896 == 896	no	OK (built + inference correct)
Qwen2.5-1.5B-Instruct	qwen2	128	1536 == 1536	no	OK (decode engine 5891 MB)
Qwen3-0.6B	qwen3	128	2048 != 1024	yes	FAIL
Qwen3-1.7B	qwen3	128	2048 == 2048	yes	FAIL (identical error)

Ruled out as the cause:

• sm_121 in general — TensorRT compiles engines fine on this GPU. Qwen3
  prefill engines build (1145 MB), and both Qwen2.5 models build and run.
• head_dim = 128 — Qwen2.5-1.5B has head_dim 128 and builds.
• heads * head_dim != hidden_size — Qwen3-1.7B is symmetric (2048 == 2048)
  and still fails.
• QK-norm — setting q_norm/k_norm to None in both decode paths
  (dual_profile_decoder_builder.py:689,695 and graph_blocks.py:210,215)
  changes nothing; both errors reproduce unchanged.
• Attention IO form — adding
  attention.query_form = attention.key_value_form = trt.AttentionIOForm.PADDED_BHND
  at families/qwen/graph_ops.py:1260 does not help. (Tried because SM120/121
  support only kPADDED_BHND, and families/sam2/image_builder.py:141-142 is
  the only place in the repo that sets this explicitly, while the Qwen native
  path relies on the default.)

Qwen3-0.6B additionally fails with --precision fp16, --max-cache-length 4096,
and --recipe qwen.decode_attention_region@2 ... --graph-role decode.

Working workaround

trtmc build Qwen/Qwen3-0.6B --dynamic-kv-cache -o qwen3-0.6b.bundle
trtmc run qwen3-0.6b.bundle --prompt "What is the capital of France? Answer in one word." \
  --chat-template --no-thinking
# Generated text: Paris

--dynamic-kv-cache disqualifies native-KV routing
(build_routing.py: "native Qwen3 uses one fixed physical KV capacity") and
drops the split layout, landing on the single-engine legacy dense-mask graph in
bf16 — which compiles on sm_121 with QK-norm intact. Verified end-to-end on
Qwen3-0.6B (1516 MB engine, decode 5.9 ms) and Qwen3-1.7B (3958 MB).

Note the combination matters: --dynamic-kv-cache with --precision fp16
still fails, because fp16 trips a separate crash (below).

Why we believe this is SM120/121-specific

TensorRT's fused-attention documentation splits support into two tables
(link):

• Supported Attention Fusions on SM100, SM103, and SM110 — IO form
  kPADDED_BHND and kPACKED_NHD, no sequence-length restriction.
• Supported Attention Fusion for Other SM Versions — covers SM75-SM90,
  SM120, SM121 — IO form kPADDED_BHND only, with tighter constraints.

The repo's validated environments are all datacenter/embedded parts
(benchmarks/performance/environments/{gb300,auto-thor,l4t-thor}.yaml), and the
BYOK kernel exporter hard-gates on compute capability (10, 3)
(families/qwen/kernels/export_flashinfer_decode_attention.py:199). We found no
mention of SM120/121 anywhere in the repo.

This matches a known ecosystem pattern of missing fused-attention kernels on
GB10-class parts: NVIDIA/TensorRT-LLM#11799, sgl-project/sglang#11658,
Dao-AILab/flash-attention#1969, NVIDIA/TensorRT#4743, NVIDIA/TensorRT#4715.

We could not confirm this directly — only one GPU was available, so we cannot
test whether Qwen3 builds on SM100/103. If someone can run
trtmc build Qwen/Qwen3-0.6B on a GB300 or Thor, that would settle it.

Two related problems found along the way

1. --precision fp16 crashes with an internal assertion

fp16 disables native KV (build_routing.py: "native Qwen3 requires BF16") and
uses the legacy dense-mask graph, which crashes differently:

MyelinCheckException: wrap_attention_op_in_kgen.cpp:1909:
  CHECK(expected_live_inputs == knode->inputs().size()) failed.
  Expected 3 live inputs, got 4

This one looks like a genuine bug rather than a capability gap — it is an
internal consistency check, not a "no kernel available" message. Qwen2.5-0.5B
builds fine in fp16 on the same machine, so it is config-dependent.

2. Hardcoded 1 GB workspace is too small at the default cache length

families/qwen/dual_profile_decoder_builder.py:396 sets
set_memory_pool_limit(trt.MemoryPoolType.WORKSPACE, 1 << 30). Building
Qwen3-0.6B in fp16 at the model-default cache (40960 tokens) requests
2,598,728,448 bytes and fails:

UNSUPPORTED_STATE: Skipping tactic 0 due to insufficient memory on requested
size of 2598728448 detected for tactic 0x0000000000000000.

This host has 121 GB of unified memory, so the cap is the only limit. Raising it
to 8 GB unblocks fp16 prefill (decode still hits problem 1).

Suggested improvements

1. Fail with an actionable message. native_kv_build_capability() already
   has a clean list of disqualifying reasons. Consider adding a device-capability
   check so unsupported SMs are disqualified there — either falling back to the
   legacy graph automatically, or raising an error that names
   --dynamic-kv-cache — instead of surfacing a raw Myelin exception.
2. Document the supported SM range for the Qwen3 native KV path.
3. Derive the workspace limit from the request (or make it configurable)
   rather than hardcoding 1 GB.

Not viable as workarounds (for the record)

• attention.decomposable = True at graph_ops.py:1260: the native bf16
  split build then succeeds (prefill + decode, 43 s), but inference produces
  garbage tokens and decode is roughly 10x slower. The existing comment
  ("The prototype deliberately requires TensorRT's fused attention tactic")
  explains why this is disabled.
• BYOK / TVM-FFI kernel slot: limited to one slot per bundle, enforced in
  code at src/plugins/tvm_ffi_runtime_bindings.cpp:242
  ("kernel_slots.json.slots must contain exactly one slot"), so it cannot
  replace all 28 decode attention layers. Confirmed experimentally: applying
  --recipe moves the fused region start from layer 2 to layer 87 (one
  transformer layer carved out) but the build still fails on the rest.

What have you tried?

No response

TensorRT-Model-Connect version

c926bdc9a6771fb02b3deecda2ef030e5dbe948a

Model and revision

Qwen/Qwen3-0.6B

Environment details

No response

Submission checks

• I searched the documentation and open and closed issues first.
• This question contains no secrets, private URLs, or restricted artifacts.

Start with native_kv_build_capability and routing in build_routing.py, then inspect families/qwen/dual_profile_decoder_builder.py and graph_ops.py. Reproduce the Qwen3 command with --dynamic-kv-cache on SM121 and compare the native and legacy paths. Done should mean unsupported native-KV devices receive an actionable fallback or error, with the supported SM range documented.