Myelin NVRTC compilation failure on Jetson Thor (SM110) building Fast-FoundationStereo - reproduces on TRT 10.14.1 and 11.2.1.2
Open
Nobody has claimed this yet.
Module:Embedded
Module:Engine Build
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Title: Myelin NVRTC compilation failure on Jetson Thor (SM110) building Fast-FoundationStereo — reproduces on TRT 10.14.1 and 11.2.1.2
Environment
- Jetson Thor (SM110), JetPack 7 / L4T, unified memory 122 GB
- TensorRT 10.14.1 (JetPack tarball trtexec) and 11.2.1.2 (pip
tensorrtcu13 aarch64, python API) - Model: Fast-FoundationStereo, exported from the NGC checkpoint (
nvidia/tao/fast-foundationstereov1.2,model_best_bp2_serialize.pth) with the officialscripts/make_single_onnx.py(NVlabs/Fast-FoundationStereo, opset 17),--height 288 --width 480 --valid_iters 4, exporter run innvcr.io/nvidia/pytorch:25.06-py3-igpu
Failure (identical on both TRT versions, fp32 — no precision flags involved)
TRT 10.14.1:
Error[9]: Skipping tactic 0x0 due to exception [myelin_graph.h:1182] MyelinCheckException:
nvrtc_compile.cpp:1118: CHECK(success) failed. NVRTC Compilation failure
Error[10]: IBuilder::buildSerializedNetworkToStream: Error Code 10: Internal Error
(Could not find any implementation for node {ForeignNode[/Concat.../Unsqueeze_166]}.)
TRT 11.2.1.2:
MyelinCheckException: nvrtc_compile.cpp:1122: CHECK(success) failed. NVRTC Compilation failure
Error Code 10: Internal Error (Could not find any implementation for node {ForeignNode[/Concat.../Cast_118]}.)
Additional data points
- An older single-ONNX export of the same model (576x960, iters_8, exported ~May 2026 with an older torch) builds and runs correctly in fp32 on the same machine/TRT 10.14.1 (1199 ms) — the failure is specific to graph patterns emitted by the newer exporter/torch.
- fp16 builds of the older export segfault the TRT compiler on SM110 (10.13.3 and 10.14.1; July 2026 finding).
- Full FoundationStereo-S (NGC dynamic ONNX) at 288x480 builds fp32 on 10.14.1 (1052 ms) — so small dims per se are fine.
- Looks related to #4715 (sm_120: Myelin "no valid tactics" on Fast-FS post_runner at all precisions, fused 3D-ConvTranspose/Cast pattern).
Questions
- Is there a fix or workaround for the SM110 Myelin/NVRTC codegen failure (builder flags, tactic source restrictions, graph rewrites)?
- Is the repo's plugin path (
make_plugin_onnx.py+ cpp plugin) the intended deployment route on Thor today? - Is fp16 expected to work for this model on SM110, given the exporter's own printed recipe suggests
trtexec --fp16?
Full build logs and both ONNX files available on request.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the failure with the newer ONNX export from scripts/make_single_onnx.py using trtexec or the Python API, then compare it with the older working export. Read related issue #4715 and inspect the make_plugin_onnx.py and C++ plugin path mentioned in the report. Done means a confirmed fix or workaround, with the expected fp16 and plugin deployment behavior documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, python
- Domain
- machine-learning, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100