NVIDIA / NVIDIA/TensorRT

TensorRT engine execution return all 0 due to warning[ScatterND with reduction==‘none’ only guarantees to be correct if indices are not duplicated.]

Open
#4,425 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

triaged
Dominant language
C++
Stars
13.4k
Forks
2.4k
Avg merge
5d 3h
Merged PRs (30d)
2

Description

Greetings, everyone.

Right now I am working on this project: GitHub - facebookresearch/sam2: The repository provides code for running inference with the Meta Segment Anything Model 2 (SAM 2), links for downloading the trained model checkpoints, and example notebooks that show how to use the model.
I am using this guidance to export sam2.1 encoder[small] onnx model: GitHub - ibaiGorordo/ONNX-SAM2-Segment-Anything: Python scripts for the Segment Anythin 2 (SAM2) model in ONNX
The exported onnx file can be successfully converted to TensorRT engine file;
The hardware/software information is shown as following:
[04/20/2025-17:31:16] [I] === Device Information ===
[04/20/2025-17:31:16] [I] Available Devices:
[04/20/2025-17:31:16] [I] Device 0: “Orin” UUID: GPU-a00bb704-da56-555b-a79c-65a4e3662de8
[04/20/2025-17:31:16] [I] Selected Device: Orin
[04/20/2025-17:31:16] [I] Selected Device ID: 0
[04/20/2025-17:31:16] [I] Selected Device UUID: GPU-a00bb704-da56-555b-a79c-65a4e3662de8
[04/20/2025-17:31:16] [I] Compute Capability: 8.7
[04/20/2025-17:31:16] [I] SMs: 16
[04/20/2025-17:31:16] [I] Device Global Memory: 62840 MiB
[04/20/2025-17:31:16] [I] Shared Memory per SM: 164 KiB
[04/20/2025-17:31:16] [I] Memory Bus Width: 256 bits (ECC disabled)
[04/20/2025-17:31:16] [I] Application Compute Clock Rate: 1.3 GHz
[04/20/2025-17:31:16] [I] Application Memory Clock Rate: 1.3 GHz
[04/20/2025-17:31:16] [I]
[04/20/2025-17:31:16] [I] Note: The application clock rates do not reflect the actual clock rates that the GPU is currently running at.
[04/20/2025-17:31:16] [I]
[04/20/2025-17:31:16] [I] TensorRT version: 10.7.0
[04/20/2025-17:31:16] [I] Loading standard plugins
[04/20/2025-17:31:16] [I] [TRT] [MemUsageChange] Init CUDA: CPU +2, GPU +0, now: CPU 31, GPU 13403 (MiB)
[04/20/2025-17:31:18] [I] [TRT] [MemUsageChange] Init builder kernel library: CPU +928, GPU +749, now: CPU 1002, GPU 14197 (MiB)
[04/20/2025-17:31:18] [I] Start parsing network model.

Issue:

I have successfully loaded the engine file and feed it with data using Python;
However, the console outputs when running the inference:
2025-04-20 17:50:14.100165426 [W:onnxruntime:Default, scatter_nd.h:51 ScatterNDWithAtomicReduction] ScatterND with reduction==‘none’ only guarantees to be correct if indices are not duplicated.
2025-04-20 17:50:14.100240944 [W:onnxruntime:Default, scatter_nd.h:51 ScatterNDWithAtomicReduction] ScatterND with reduction==‘none’ only guarantees to be correct if indices are not duplicated.

Due to above issue, all values in all output tensors are 0.
Online search doesn’t show too much helpful/valuable information to solve this issue.
If you have any clue/idea about this, could you please share it with me?
Thank you very much. :)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the TensorRT 10.7 inference on the Orin using the exported SAM2.1 encoder ONNX model and Python execution path. Inspect the ScatterND warning and compare TensorRT engine outputs with the source ONNX outputs; done means identifying why the engine returns all-zero tensors and confirming nonzero, matching results.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.