microsoft / microsoft/onnxruntime

[Feature Request] Compile API: allow TensorRT-RTX compile-only sessions without a live/matching GPU

Open
#29,757 0 comments 0 reactions 0 assignees View on GitHub
ep:TensorRT feature request
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the feature request

Compiling an EPContext model via the Compile API (`OrtCompileAPI::CompileModel()`) with the TensorRT-RTX EP currently (as of v1.27.1) requires the same GPU class as inference, for two reasons:

1. **The EP constructor requires a live, matching GPU.** [`NvExecutionProvider::NvExecutionProvider`](https://github.com/microsoft/onnxruntime/blob/df2ba1cf8108aa63627cf4cdf8f807880b938616/onnxruntime/core/providers/nv_tensorrt_rtx/nv_execution_provider.cc#L954) calls `cudaSetDevice()`/`cudaGetDeviceProperties()` and fails unless the GPU's compute capability is 8.6, 8.9, or ≥12.0, even for a compile-only session that will never run inference.

2. **The engine is always built for [`ComputeCapability::kCURRENT`](https://github.com/microsoft/onnxruntime/blob/df2ba1cf8108aa63627cf4cdf8f807880b938616/onnxruntime/core/providers/nv_tensorrt_rtx/nv_execution_provider.cc#L2753).** Since it isn't exposed as a provider option, the compiled artifact is always locked to the GPU that built it. There's no way to get TensorRT-RTX's own default: a portable multi-architecture engine, or an explicitly chosen target.

Neither is a TensorRT-RTX limitation: the SDK's `tensorrt_rtx` CLI and builder API both support building with [no live GPU at all](https://docs.nvidia.com/deeplearning/tensorrt-rtx/latest/inference-library/cpu-engines.html) and explicit `--computeCapabilities=` targeting.

**Request:** allow the Compile API path to construct the EP without a live/matching GPU (extending the compile-only handling from #28503), and provide a way to control the target compute capability instead of unconditionally locking to `kCURRENT`.

**Related:**

- #26663: introduced the `ComputeCapability::kCURRENT` setting discussed in point 2. It is a sensible default when building and running on the same device, which we're not asking to change.
- #28503: made compile-only sessions skip the final `deserializeCudaEngine()`/`createExecutionContext()` calls; the device checks in the EP constructor are not covered by that change.
- #27801: not directly related, but mentions a limitation regarding the compatible compute capabilities (hardware the SDK documents as supported being excluded by the allowlist).

**N.B.** The standalone [NVIDIA/TensorRT-RTX-EP-ABI](https://github.com/NVIDIA/TensorRT-RTX-EP-ABI) plugin (the built-in EP's designated successor) seems to have the same device-gate and `kCURRENT` behavior.

### Describe scenario use case

CI/build-farm workflows: pre-compiling EPContext models for a fleet of deployment machines with heterogeneous GPUs (those with compute capability < 8.6 falling back to other EPs), on a build machine without the target GPU, or even without any GPU at all. This is the exact scenario the TensorRT-RTX docs support at the SDK level, but which the Compile API rejects.

Contributor guide

Open the contributing guide

Research direction

Start in onnxruntime/core/providers/nv_tensorrt_rtx/nv_execution_provider.cc, especially NvExecutionProvider and the Compile API path through OrtCompileAPI::CompileModel(). Review the compile-only handling from issue #28503 and the current ComputeCapability::kCURRENT use. Done means compile-only sessions can work without a live or matching GPU and can select an explicit or portable compute-capability target.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
backend, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.