[Bug][Jetson AGX Orin] TensorRT slot.cpp decode assertion on every request after rebuilding at maxBatchSize=1 (fc_h_fusion workaround from #151)

Describe the bug

Possibly related regression surfaced immediately after applying the -peep:fc_h_fusion=off workaround discussed in #151.

Environment

• Device: Jetson AGX Orin 64GB Developer Kit
• Jetson Linux / L4T: R39 (39.2.0)
• Kernel: 6.8.12-1021-tegra
• CUDA: 13.2.1, Driver 595.58.03
• TensorRT: 10.16.2.10
• TensorRT Edge-LLM: v0.9.1, commit 7f061f21f0a581ba234a1e233c9315b89d8e47d6
• Model: Qwen3.6-35B-A3B-GPTQ-Int4 (hybrid Mamba/GDN + Attention MoE, INT4 quantized), head_dim=256, 40 layers (30 linear-attn / Mamba, 10 full attention), max_input_len=8192, max_kv_cache_capacity=32768

History

The originally deployed engine was built with --maxBatchSize 2. Per #151's root-cause analysis, applyMyelinCompileWorkarounds() in cpp/builder/llmBuilder.cpp only appends -peep:fc_h_fusion=off when maxBatchSize == 1, so that build never got the workaround. It hit a hard CUDA illegal-memory-access repeatedly under real (not synthetic) inference load:

RuntimeError: CUDA runtime error in cudaMemsetAsync(recurrentDst, 0, recurrentBatchBytes, stream): an illegal memory access was encountered
RuntimeError: CUDA runtime error in cudaMemsetAsync(convDst, 0, convBatchBytes, stream): an illegal memory access was encountered

Confirmed via dmesg as a genuine GPU-level fault (not just an application error):

nvgpu: 17000000.gpu gv11b_mm_mmu_fault_handle_buf_valid_entry:530 [ERR] page fault error: err_type = 0x8, fault_status = 0x200
nvgpu: 17000000.gpu gv11b_fb_mmu_fault_info_dump:278 [ERR] [MMU FAULT] ... fault type: invalid pde, access type: virt read
nvgpu: 17000000.gpu ga10b_fifo_ctxsw_timeout_isr:354 [ERR] Host pfifo ctxsw timeout error

On one occurrence, the driver's own automatic TSG/channel teardown recovery also failed (nvgpu_tsg_unbind_channel:1010 [ERR] Channel 2040 unbind failed, tearing down TSG 1), leaving the GPU channel stuck and requiring a full host reboot to clear -- a plain container/process restart was not sufficient to recover once that happened.

Reproducing the fix

Rebuilt the exact same already-exported/quantized ONNX checkpoint with --maxBatchSize 1 (all other flags unchanged):

llm_build --onnxDir <onnx-production/llm> --engineDir <engine-batch1> \
  --maxInputLen 8192 --maxKVCacheCapacity 32768 --maxBatchSize 1

Confirmed in the build log that the workaround now applies:

Using __LUNOWUD=-mlir:autotune:num_threads=1 -mlir:collective:fp4=off -cask_fusion:async_policy=1 -peep:fc_h_fusion=off

(vs. the batch=2 build's log, which omitted -peep:fc_h_fusion=off entirely.)

The original illegal-memory-access did not recur after this rebuild.

New bug

Every single inference request now fails consistently and immediately (no exceptions observed across dozens of requests over several minutes), right after the runtime flaps the optimization profile on each request:

[INFO] [TensorRT] Switching optimization profile from: 1 to 0. Please ensure there are no enqueued operations pending in this context prior to switching profiles
[INFO] [TensorRT] Switching optimization profile from: 0 to 1. Please ensure there are no enqueued operations pending in this context prior to switching profiles
[ERROR] [TensorRT] [slot.cpp::decode::47] Error Code 2: Internal Error (Assertion slots failed. encoded reference to slot found, but slots missing In decode at /_src/runtime/core/slot.cpp:47)

Followed by, at the experimental/server/api_server.py level:

RuntimeError: Failed to execute base model for prefill step.

Notably:

• No fresh kernel-level GPU fault in dmesg this time -- this is a softer, TensorRT-internal assertion failure, not a hardware channel corruption like the original bug. In that sense it "fails closed" rather than corrupting GPU state, which is an improvement, but it is 100% reproducible and fully blocking.
• The 1 -> 0 -> 0 -> 1 profile-switch flapping happens on every request, not once at startup, which was not the pattern observed on the batch=2 build (which switched profiles once at load time).
• slot.cpp is not part of the public repo (not under cpp/), so I cannot inspect or bisect it directly.
• Server startup correctly reports maxBatch=1 and Runtime batch size set to: 1 (from engine bundle), so the engine itself built and loaded as expected.

Expected behavior

A request should complete successfully once maxBatchSize=1 has been applied and the fc_h_fusion workaround is confirmed active in the build log.

Actual behavior

100% of requests fail with the slot.cpp::decode assertion above.

Additional notes

Happy to provide the full build log, engine config.json, or compute-sanitizer output if that's useful -- let me know what would help triage. This is currently fully blocking inference on this deployment. @nvluxiaoz

Start with cpp/builder/llmBuilder.cpp and the maxBatchSize/workaround behavior described in the issue, then inspect experimental/server/api_server.py and the engine config. Rebuild the exported ONNX checkpoint with maxBatchSize=1 and compare the build and profile-switch logs. Done means requests complete without the slot.cpp decode assertion or the prefill failure.