SAM3Image inference latency remains ~1.1s on RTX 4090 with TensorRT EP — is this expected? Request for an official high-performance TensorRT deployment path

## Summary

We are benchmarking **SAM3 (Sam3Image)** inference on high-end NVIDIA GPUs and would like to clarify whether the observed latency with TensorRT is expected, and whether there is (or will be) an **official high-performance TensorRT deployment path** for SAM3.

Using **ONNXRuntime with TensorRT Execution Provider**, end-to-end latency for a single **~600×500 image** on an **RTX 4090** remains around **~1.1s per image (after warm-up)**.
This is only about **20% faster than CUDA EP** on the same machine (~1.4s), which seems low given the hardware and TensorRT’s typical gains.

TensorRT is clearly active (engine build and serialization logs are present), but the overall speedup is modest.

---

## Environment

- **OS**: Ubuntu 24.04
- **GPU**: NVIDIA GeForce RTX 4090 (24GB)
- **Driver**: 580.126.09
- **CUDA**: 13.0 (`nvcc 13.0.88`)
- **TensorRT**: 10.14.1.48 (CUDA 13 build)
- **ONNXRuntime**: official prebuilt binaries
- **Precision**: FP32
- **Image resolution**: ~600×500 JPEG

---

## Inference Configuration

- Model: **Sam3Image**
- Multiple text prompts batched in a single forward pass
(`independent = false`)
- TensorRT enabled via ONNXRuntime TensorRT EP
- Warm-up performed before measurement
- Observed latency (single image):
- **~1.1s / image**
- CUDA EP baseline on same machine: ~1.4s

---

## Observation

Although TensorRT EP is enabled and active, the performance improvement is relatively small (~20%).

This suggests that inference may still be dominated by factors such as:

- limited graph partitioning / fusion when using ORT + TensorRT EP
- attention or decoder stages that do not map efficiently to TensorRT
- dynamic shape overhead
- preprocessing / postprocessing not fully accelerated
- architectural bottlenecks within SAM3 itself

This raises the question of whether **ONNXRuntime + TensorRT EP is intended as a high-performance deployment path for SAM3**, or primarily as a compatibility option.

---

## Questions

1. **Is ~1.1s per ~600×500 image on RTX 4090 expected for Sam3Image today?**
2. If not, what is the **recommended way to optimize inference latency**?
3. Are there plans to provide an **official TensorRT-optimized deployment path**, such as:
- native TensorRT engine export (not via ORT EP)
- static or semi-static shape TensorRT engines
- FP16 / BF16 / INT8 reference configurations
- additional operator fusion or decoder optimizations
4. Are there known architectural components in SAM3 (e.g. attention, mask decoding) that currently limit TensorRT speedup?

---

## Motivation

SAM3 is a strong foundation model for many real-time and near-real-time vision applications.
For production use, however, **sub-second inference on high-end GPUs is critical**.

An **official, documented TensorRT deployment path** (even if constrained to specific shapes or modes) would significantly improve deployability and reduce duplicated optimization efforts across users.

We would be happy to help validate or benchmark any recommended configuration.

---

Thanks for the great work on SAM3 — looking forward to guidance on high-performance inference.

The issue names no repository files, tests, or entry points. Begin by reproducing the described warm-up benchmark for Sam3Image with ONNXRuntime's TensorRT and CUDA execution providers, then compare the results with the reported timings. Done would require a maintainer-confirmed expectation, recommended deployment path, or concrete optimization target.

Written by the indexing model from the issue text.