Bug: TensorRT 11.2 bundles can hit Myelin CUDA illegal memory access after successful generation

This was generated by AI during triage.

Summary

TensorRT 11.2.0.113 bundles can complete prefill/decode and then report CUDA error 700 / illegal memory access while Myelin unloads the CUDA module. The process exits with a segmentation fault.

This was discovered while validating #245. In a cache-length-1741 sweep on GB300, 16 of 33 models reproduced the runtime signature.

Related issue: #245

Environment

• TRTMC source: PR #413, commit a10ba64820082ad3d21dfc67bffe444dd95c9626
• TensorRT: 11.2.0.113, ABI 11.2
• CUDA image: 13.3
• GPU: NVIDIA GB300, physical GPU 1
• Host driver: 595.58.03
• Forward-compat libcuda: 610.43.02
• Test image: trtmc-issue245-yizhuo-trt112:a10ba648
• Cache length: 1741
• TRT 11.2 SDK artifact SHA-256: 3d09048311c7bf8f6d32b3e3cfe2c615de9397e9289f4d436277dd35cb58c3dd

Minimal reproduction

Run inside a TRTMC TensorRT 11.2 / CUDA 13.3 environment. No task-eval suite or external dataset is required.

Build the bundle:

./build/trtmc build google/gemma-2-2b-it \
  -o /tmp/gemma-2-2b-c1741.trtfb \
  --max-cache-length 1741 \
  --precision fp16

Observed successful build:

[trtmc build] Building split decoder engines (cache=1741) ...
[trtmc build] Prefill engine built [118.4s] (6118.5 MB)
[trtmc build] Decode engine built [114.6s] (6115.2 MB)
[trtmc build] Bundle saved: /tmp/gemma-2-2b-c1741.trtfb

Run the bundle directly with a one-token prompt:

./build/trtmc run /tmp/gemma-2-2b-c1741.trtfb \
  --prompt "Hello" \
  --max-new-tokens 1 \
  --temperature 1.0 \
  --top-p 1.0 \
  --top-k 1 \
  --seed 245

This standalone command reproduced twice with two different short prompts; both runs exited with status 139.

Actual behavior

The pipeline loads and completes prefill/decode. The failure is surfaced during Myelin graph/module teardown:

[trtmc] KV cache rows=1741 (bundle max=1741, row=106496 B, cache=176.82 MiB, state=176.92 MiB)
[trtmc] Pipeline loaded (strategy=gemma_decoder_kv_cache, backend=trt_new_runtime)
[trtmc.timing] prefill_ms=92.179303 decode_ms=0.350403 total_ms=92.529706
[trt] ERROR: ... CUDA error 700 when unloading module ...
Internal Error: MyelinCheckException: l2cm_layer_internal.cpp:109: CHECK(false) failed.
error: cuModuleUnload(module_) failed with error an illegal memory access was encountered
!!!!!!! Segfault encountered !!!!!!!
  File "<unknown>", line 0, in cuStreamSynchronize
  File "<unknown>", line 0, in trtmc::TrtModuleImpl::~TrtModuleImpl()
  File "<unknown>", line 0, in trtmc::GemmaTextGenerationPipeline::~GemmaTextGenerationPipeline()
  File "<unknown>", line 0, in (anonymous namespace)::cmd_run(trtmc::cli::CliArgs const&)

The standalone trtmc run process exits with status 139.

Expected behavior

The bundle should complete generation and destroy all TensorRT/Myelin/CUDA resources without CUDA error 700, illegal memory access, MyelinCheckException, or a segmentation fault.

Sweep results

All 33 models reached TensorRT runtime with TensorRT 11.2 bundles. Runtime classification was based on the raw runtime log, not only the top-level task-eval return code, because some VLM runs returned successfully while still logging the illegal-access/Myelin signature.

• Runtime-safety pass: 17 models
• Runtime-safety failure: 16 models
• Build-time #245 signature: 0 models

16 models with the runtime-safety failure

• gemma-2-2b
• nemotron-nano-4b
• phi-moe
• phi3-mini
• qwen3-moe-30b-a3b
• qwen35-9b
• deepseek-v2-lite
• falcon3-1b
• minitron-4b-width
• mixtral-stories-15m
• olmo-1b
• olmo2-1b
• stablelm2-1.6b
• internvl3-2b
• internvl3-8b
• qwen3-vl-2b

Passing controls at the same cache length include qwen3-0.6b-fp16 (10/10 samples with clean teardown), mistral-7b, deepseek-v2-tiny, gpt2-125m, and qwen25vl-3b.

Impact

• This is a latent GPU memory-safety failure: valid output may be produced before CUDA reports the error.
• A successful prediction or a zero top-level task-eval return code is not sufficient to classify the runtime as safe.
• The affected set spans multiple model strategies and text/VLM paths, so it is not isolated to Gemma.
• The consistent delayed detection at cuModuleUnload suggests the out-of-bounds access occurs earlier during execution and becomes visible at synchronization/module teardown.

Suggested regression criteria

• The Gemma-2-2B reproduction completes with a clean teardown and no CUDA/Myelin safety signature.
• The 16 affected model entries complete without CUDA error 700, illegal memory access, or l2cm_layer_internal.cpp:109.
• Runtime validation scans the full raw log after generation, including teardown, rather than relying only on generated output or the top-level return code.
• The original #245 Qwen3-0.6B cache=2048 build remains passing.

Reproduce the failure with the documented ./build/trtmc build and run commands, then inspect cmd_run and the teardown paths named in the stack trace, including TrtModuleImpl and GemmaTextGenerationPipeline destructors. Compare raw runtime logs for the affected models and controls. Done means clean generation and teardown with no CUDA error 700, MyelinCheckException, illegal access, or segfault, while #245 remains passing.