TensorRT engine execution return all 0 due to warning[ScatterND with reduction==‘none’ only guarantees to be correct if indices are not duplicated.]
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 13.4k
- Forks
- 2.4k
- Avg merge
- 5d 3h
- Merged PRs (30d)
- 2
Description
Greetings, everyone.
Right now I am working on this project: GitHub - facebookresearch/sam2: The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.
I am using this guidance to export sam2.1 encoder[small] onnx model: GitHub - ibaiGorordo/ONNX-SAM2-Segment-Anything: Python scripts for the Segment Anythin 2 (SAM2) model in ONNX
The exported onnx file can be successfully converted to TensorRT engine file;
The hardware/software information is shown as following:
[04/20/2025-17:31:16] [I] === Device Information ===
[04/20/2025-17:31:16] [I] Available Devices:
[04/20/2025-17:31:16] [I] Device 0: “Orin” UUID: GPU-a00bb704-da56-555b-a79c-65a4e3662de8
[04/20/2025-17:31:16] [I] Selected Device: Orin
[04/20/2025-17:31:16] [I] Selected Device ID: 0
[04/20/2025-17:31:16] [I] Selected Device UUID: GPU-a00bb704-da56-555b-a79c-65a4e3662de8
[04/20/2025-17:31:16] [I] Compute Capability: 8.7
[04/20/2025-17:31:16] [I] SMs: 16
[04/20/2025-17:31:16] [I] Device Global Memory: 62840 MiB
[04/20/2025-17:31:16] [I] Shared Memory per SM: 164 KiB
[04/20/2025-17:31:16] [I] Memory Bus Width: 256 bits (ECC disabled)
[04/20/2025-17:31:16] [I] Application Compute Clock Rate: 1.3 GHz
[04/20/2025-17:31:16] [I] Application Memory Clock Rate: 1.3 GHz
[04/20/2025-17:31:16] [I]
[04/20/2025-17:31:16] [I] Note: The application clock rates do not reflect the actual clock rates that the GPU is currently running at.
[04/20/2025-17:31:16] [I]
[04/20/2025-17:31:16] [I] TensorRT version: 10.7.0
[04/20/2025-17:31:16] [I] Loading standard plugins
[04/20/2025-17:31:16] [I] [TRT] [MemUsageChange] Init CUDA: CPU +2, GPU +0, now: CPU 31, GPU 13403 (MiB)
[04/20/2025-17:31:18] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +928, GPU +749, now: CPU 1002, GPU 14197 (MiB)
[04/20/2025-17:31:18] [I] Start parsing network model.
Issue:
I have successfully loaded the engine file and feed it with data using Python;
However, the console outputs when running the inference:
2025-04-20 17:50:14.100165426 [W:onnxruntime:Default, scatter_nd.h:51 ScatterNDWithAtomicReduction] ScatterND with reduction==‘none’ only guarantees to be correct if indices are not duplicated.
2025-04-20 17:50:14.100240944 [W:onnxruntime:Default, scatter_nd.h:51 ScatterNDWithAtomicReduction] ScatterND with reduction==‘none’ only guarantees to be correct if indices are not duplicated.
Due to above issue, all values in all output tensors are 0.
Online search doesn’t show too much helpful/valuable information to solve this issue.
If you have any clue/idea about this, could you please share it with me?
Thank you very much. :)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the TensorRT 10.7 inference on the Orin using the exported SAM2.1 encoder ONNX model and Python execution path. Inspect the ScatterND warning and compare TensorRT engine outputs with the source ONNX outputs; done means identifying why the engine returns all-zero tensors and confirming nonzero, matching results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100