tensorrt-edgellm-build --dense nvfp4-qdq SIGSEGVs in a libnvinfer compiler worker thread on Jetson Thor (CC 11.0) for Cosmos-Reason2-2B; --dense fp16 builds fine from the same NVFP4 checkpoint

Describe the bug

Impact: blocker. NVFP4 is the reason we adopted TensorRT Edge-LLM on this platform; falling back
to FP16 removes the benefit entirely.

Building the LLM component of an NVFP4-quantized Cosmos-Reason2-2B checkpoint crashes with SIGSEGV
during engine build. There is no TensorRT ERROR, no assertion and no exception — the process dies
silently shortly after the builder enters the compiler backend.

Key observations:

• The same checkpoint builds successfully with --dense fp16.
• The visual component (FP8, per hf_quant_config.json) builds successfully (~70.8 s, 416 MB).
• Only --dense nvfp4-qdq with --components llm crashes.
• The front-end correctly detects the checkpoint as NVFP4:
  model_type=qwen3_vl_text layers=28 hidden=2048 experts=0 quant=nvfp4
  — so this is not a checkpoint-detection problem.

Verbose logs show the build progressing normally through graph construction (3362 layers),
Q/DQ optimization, LayerNorm pattern matching and Myelin relocation, ending at
After Myelin optimization: 1 layers, then autotuning / Timing Runner — and then the process dies.

Crash appears to be in a multi-threaded compiler worker

The crash occurs in one of the worker threads spawned immediately after
[TRT] Compiler backend is used during engine build.
(timestamps and LWP ids elided below for brevity; full output is in the attached gdb_bt2.txt):

INFO experimental.builder.core.builder: [TRT] Compiler backend is used during engine build.
[New Thread 0xfffd5fa1f180 (LWP ...)]
[Thread 0xfffd5fa1f180 (LWP ...) exited]
[New Thread 0xfffd5fa1f180 (LWP ...)]
[New Thread 0xfffd5f20f180 (LWP ...)]
[New Thread 0xfffd5e9ff180 (LWP ...)]
[New Thread 0xfffd5e1ef180 (LWP ...)]
[New Thread 0xfffd5d9df180 (LWP ...)]
[New Thread 0xfffd5d1cf180 (LWP ...)]
[New Thread 0xfffd5c9bf180 (LWP ...)]

Thread 21 "python" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0xfffd5e9ff180 (LWP ...)]
0x0000ffffb65f16a0 in ?? () from /lib/aarch64-linux-gnu/libnvinfer.so.10
#0  0x0000ffffb65f16a0 in ?? () from /lib/aarch64-linux-gnu/libnvinfer.so.10
#1  0x0000fffd5e9f58f8 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

At the moment of the fault, the sibling compiler threads are inside allocator / lock paths of the
same library, which may point at heap corruption or a data race rather than a plain null deref:

Thread 25:
#0  __pthread_rwlock_rdlock_full64 (...) at ./nptl/pthread_rwlock_common.c:298
#1  ___pthread_rwlock_rdlock (rwlock=0x...) at ./nptl/pthread_rwlock_rdlock.c:26
#2  0x0000ffffb548d3c8 in ?? () from /lib/aarch64-linux-gnu/libnvinfer.so.10

Thread 24:
#1  __GI___libc_malloc (bytes=160) at ./malloc/malloc.c:3347
#2  operator new(unsigned long) () from /lib/aarch64-linux-gnu/libstdc++.so.6
#3  0x0000ffffb42c1670 [PAC] in ?? () from /lib/aarch64-linux-gnu/libnvinfer.so.10
#4  0x0000ffffb42ccddc in ?? () from /lib/aarch64-linux-gnu/libnvinfer.so.10
#5  0x0000ffffb5d912b4 in ?? () from /lib/aarch64-linux-gnu/libnvinfer.so.10

Thread 23:
#2  __GI___libc_malloc (bytes=16) at ./malloc/malloc.c:3320
#3  0x0000ffffb52b1200 in ?? () from /lib/aarch64-linux-gnu/libnvinfer.so.10

Frames past #0/#1 in the faulting thread are unrecoverable (corrupt stack), and the shipped
libnvinfer.so.10 has no symbols, so we cannot localize further from our side.

Question: is there a way to force single-threaded engine compilation (builder worker count /
tactic timing threads) so the crash can be localized to a specific layer or tactic? If such a knob
exists we are happy to re-run and attach the result.

Control experiments — both confirmed working

Two independent controls rule out a bad checkpoint and rule out a platform-level NVFP4 problem:

1. NVFP4 kernels do run on this device. An NVFP4 engine for the same model, produced through the
   quantize → ONNX → TensorRT path, loads and runs inference correctly with llm_inference on this
   same Jetson Thor. NVFP4 execution on this GPU is therefore not the problem — only the native
   (ONNX-less) builder path is.

2. The checkpoint is healthy. Pairing the control --dense fp16 llm.engine with the
   visual.engine built from the same NVFP4 checkpoint produces correct, coherent inference output.
   The quantized weights load and dequantize correctly; this is not a corrupt quantization.

Combined with --dense fp16 and --components visual both building successfully, the failure is
isolated to the --dense nvfp4-qdq path in the native builder.

Ruled out

Hypothesis	Evidence against
Corrupt / invalid NVFP4 quantization	safetensors show correct U8 + F8_E4M3 scale layout; FP16 engine from the same checkpoint infers correctly
NVFP4 unsupported on this GPU	ONNX-path NVFP4 engine for the same model runs correctly on this device
Host OOM during build	free -h: 122 GiB total, 13 GiB used, 108 GiB available, swap 0 B; dmesg shows no OOM-killer, no Xid, no segfault entries
Shape / KV-cache size dependence	identical crash at --max-input-len 512 --max-kv-cache-capacity 1024
TensorRT too old (cf. #192, reported fixed in 10.16.1.11)	we are on 10.16.2.10, newer than the version that fixed #192
Locally modified sources	working tree clean at e8b2952 (untracked log/JSON files only)

Additional finding — --externalize-weights nvfp4_tp records nothing

Without the flag, the builder logs:

Externalizing int4_ffn, int4_moe, nvfp4_moe, nvfp4_tp, lm_head, fp16, embedding
Recorded 114 runtime checkpoint binding(s) (source layouts: fp16=114; destination dtypes: F16=114)

Note there are zero NVFP4-layout bindings even though the model is detected as quant=nvfp4.
The successful --dense fp16 build and the FP8 visual build (fp16=104) log the same shape.

With --externalize-weights nvfp4_tp explicitly:

Recorded 0 runtime checkpoint binding(s) (source layouts: ; destination dtypes: )

So no weight matches the nvfp4_tp category, and the crash is unchanged. The NVFP4 weights instead
appear embedded in the graph as constants (CONSTANT w_fp4_*, w_scales_*, ws2_*,
act_scale16/32_*), feeding a repeated DYNAMIC_QUANTIZE → Dequantize(SCALE) → MATRIX_MULTIPLY
pattern across all 28 layers.

This may or may not be related to the crash, but it looks unintended. Is nvfp4_tp expected to
match anything for a dense (non-MoE, non-TP) model?

Not a duplicate of #192

Issue #192 (Qwen3.8-27B NVFP4, Jetson Thor) superficially looks similar — Thor, NVFP4, Myelin — but
the failure differs on four material points:

	#192	This report
Failure mode	Explicit Error Code 1: Internal Error, clean abort	Silent SIGSEGV, no ERROR, no assertion
Stop location	Myelin rewrite pass removeEmptyProducersFromSubgraph (removeEmptyTensors.cpp)	Past After Myelin optimization: 1 layers, in a compiler worker thread during autotuning
TensorRT	10.13.3.9 / 10.14.1.48 — reported fixed in 10.16.1.11	10.16.2.10, newer than the version that fixed #192
Model / path	Qwen3.8-27B, linear attention: gated_delta_net ×48 + causal_conv1d ×48 (PLUGIN_V3), MTP / --specBase	Cosmos-Reason2-2B (Qwen3-VL), dense attention, no GDN plugins, no speculative decoding

The root cause identified in #192 is the 0-dim optimization profile for kSpecVerifyPhaseMarker set
in setupLinearAttentionSpecVerifyProfiles (cpp/builder/llmBuilder.cpp). That path is only reached
for linear-attention models with speculative decoding enabled. Neither applies here, so that function
is never invoked in our build.

Steps/Code to reproduce bug

1. Quantize Cosmos-Reason2-2B to NVFP4 with tensorrt-edgellm-quantize — succeeds.
   The resulting hf_quant_config.json reports quant_algo: "MIXED_PRECISION" (attached):

   • model.visual.* → FP8
   • model.language_model.layers.0..27 → NVFP4, group_size: 16
   • lm_head → NVFP4, group_size: 16

2. Verify the checkpoint really is NVFP4 (it is). model.safetensors dtype distribution:
   F32: 602, F16: 325, F8_E4M3: 301, U8: 197.
   Layer 0 shows the expected packed layout, e.g.
   *_proj.weight: U8, *_proj.weight_scale: F8_E4M3, *_proj.weight_scale_2: F32,
   with down_proj.weight at half logical width (2 FP4 values per byte).

3. Build the LLM component from that checkpoint with --dense nvfp4-qdq → SIGSEGV.

Build configuration:

# No explicit toolchain file / EMBEDDED_TARGET / TRT_PACKAGE_DIR were used.
# Native build on the Jetson Thor device itself. From build/CMakeCache.txt:
#
#   BUILD_PYTHON_BINDINGS:BOOL=ON
#   CMAKE_BUILD_TYPE:STRING=Release
#   CMAKE_GENERATOR:INTERNAL=Unix Makefiles
#   pybind11_DIR:UNINITIALIZED=<repo>/venv/lib/python3.12/site-packages/pybind11/share/cmake/pybind11
#
# CMAKE_CUDA_ARCHITECTURES is not present in the cache (never set explicitly).
# Plugin library loaded at build time: build/libNvInfer_edgellm_plugin.so

cmake .. -DCMAKE_BUILD_TYPE=Release \
         -DBUILD_PYTHON_BINDINGS=ON \
         -Dpybind11_DIR=<repo>/venv/lib/python3.12/site-packages/pybind11/share/cmake/pybind11

Runtime command used:

# FAILS with SIGSEGV
tensorrt-edgellm-build \
  --model-dir /path/to/cosmos-reason2-2b-nvfp4 \
  --engine-dir /path/to/engines/nvfp4 \
  --dense nvfp4-qdq \
  --components llm \
  --max-input-len 2048 --max-kv-cache-capacity 4048 --max-batch-size 1

# Same crash point, also reproduced with:
#   --max-input-len 512 --max-kv-cache-capacity 1024      (rules out shape/size dependence)
#   --externalize-weights nvfp4_tp                        (see "Additional finding")
#   tensorrt-edgellm-serve on the same checkpoint         (its internal build)

# SUCCEEDS (control 1): same checkpoint, FP16 dense -> llm.engine (~3.46 GB)
tensorrt-edgellm-build \
  --model-dir /path/to/cosmos-reason2-2b-nvfp4 \
  --engine-dir /path/to/engines/ctrl-fp16 \
  --dense fp16 \
  --components llm \
  --max-input-len 2048 --max-kv-cache-capacity 4048 --max-batch-size 1

# SUCCEEDS (control 2): visual component (FP8) -> visual.engine (~416 MB) in ~70.8 s
tensorrt-edgellm-build \
  --model-dir /path/to/cosmos-reason2-2b-nvfp4 \
  --engine-dir /path/to/engines/nvfp4 \
  --components visual

Expected behavior

--dense nvfp4-qdq should produce a working llm.engine, as it does for --dense fp16 on the same
checkpoint. At minimum, if this NVFP4 graph is unsupported on this platform, the builder should emit
a diagnostic error rather than segfaulting.

System information (Edge Device)

• Platform (e.g., NVIDIA Jetson Thor): NVIDIA Jetson AGX Thor (nvidia-smi reports "NVIDIA Thor")
• Software release (e.g., JetPack 7.1): JetPack 7.2.1-b49 / L4T R39 REVISION 2.1 (GCID 46758480)
• CPU architecture: aarch64 (Ubuntu 24.04.4 LTS, kernel 6.8.12-1021-tegra)
• GPU compute capability (e.g., SM110 for Jetson Thor): compute capability 11.0
  (note: the build log prints Compiled XQA kernel for SM101 — see "Other details" below)
• Total device memory: 122 GiB (13 GiB used, 108 GiB available at build time, swap 0 B)
• Build type (e.g., Release, Debug): Release
• Library versions:
  • TensorRT Edge-LLM version or commit hash: 0.10.1, commit e8b2952 (clean working tree)
  • CUDA: 13.2 (V13.2.86, cuda_13.2.r13.2/compiler.37953736_0)
  • TensorRT: 10.16.2.10-1+cuda13.2 (apt); Python binding reports 10.16.2.10
  • C++ compiler (e.g., GCC 11.4): GCC 13.3.0 (Ubuntu 13.3.0-6ubuntu2~24.04.1)
• CMake options used:
  • CMAKE_TOOLCHAIN_FILE: not set (native build on the device, not cross-compiled)
  • EMBEDDED_TARGET: not set
  • TRT_PACKAGE_DIR: not set (system TensorRT from apt was used)
• Any other details that may help:
  • Driver 595.78. Python 3.12.3 (venv). cuDNN 9.20.0.46-1 (libcudnn9-cuda-13).
  • Model: nvidia/Cosmos-Reason2-2B, quantized to NVFP4 with tensorrt-edgellm-quantize.
  • SM101 vs CC 11.0: the build logs Detected CUDA device 0 with compute capability 11.0 but
    then Compiled XQA kernel for SM101, ... with NVRTC (106680 bytes, 164 ms). Under CUDA 13,
    SM101 was renamed to SM110 and SM101 is no longer valid, so this naming looks stale.
    We do not believe this is the cause — the same SM101 line appears in the successful
    --dense fp16 build and the NVRTC compilation itself succeeds. Reporting it only in case it
    indicates a wider architecture-naming inconsistency. Relatedly, since we never set
    CMAKE_CUDA_ARCHITECTURES, please advise if an explicit value is required for correct NVFP4
    kernel selection on Thor — we are happy to rebuild and retest.
  • Possibly relevant known limitation from the docs: "TensorRT 10.15 may cause accuracy degradation
    with NVFP4 for some models; use TensorRT 10.13.3.9 shipped with JetPack 7.1 instead."
  • Attached: gdb_bt2.txt (backtrace incl. thread apply all bt),
    llm_verbose.log (full --verbose build log up to the crash),
    hf_quant_config.json (quantization configuration).

gdb_bt2.txt
hf_quant_config.json
llm_verbose.log

Start by reproducing the tensorrt-edgellm-build --dense nvfp4-qdq --components llm command and compare it with the working FP16 build. Inspect the native builder path, including cpp/builder/llmBuilder.cpp, and determine whether compiler worker or timing-thread configuration can isolate the fault. Done means the NVFP4 build succeeds or reports a diagnostic error instead of SIGSEGV.