[Bug] RetinaNet+ONNXRuntime Reference (r5.0-dev): CUDA Error (misaligned/Xid 31) in MultiStream/Offline & ValueError (NaN iscrowd) in Accuracy

Environment:

• OS: Ubuntu 20.04.6 LTS
• GPU Model: NVIDIA A800 80GB PCIe
• NVIDIA Driver Version: 530.30.02
• CUDA Toolkit Version: 12.1 (V12.1.66)(Installed via system package or runfile)
• cuDNN Version:8.9.7 (Manually installed in user space)
• ONNX Runtime Version: 1.18.0 (GPU version, installed via pip in conda env ml_perf)
• Python Version: 3.10.16 (via Miniconda environment ml_perf)
• pycocotools Version: 2.0.8 (installed via pip)
• MLPerf Inference Repo Version: Branch/Tag r5.0-dev (Cloned/managed via MLCommons CM)
• MLCommons CM Version: mlc-scripts 1.0.2 / mlcflow 1.0.10

Command Used to Reproduce:

The primary command intended to run all scenarios in valid mode:

mlcr run-mlperf,inference,_full,_r5.0-dev,_all-scenarios \
   --model=retinanet \
   --implementation=reference \
   --framework=onnxruntime \
   --category=edge \
   --execution_mode=valid \
   --device=cuda \
   --quiet

Commands used for debugging specific scenarios (after clearing results cache):

# For SingleStream (Performance runs, Accuracy fails)
mlcr run-mlperf,inference,_full,_r5.0-dev \
   --model=retinanet \
   --implementation=reference \
   --framework=onnxruntime \
   --category=edge \
   --scenario=SingleStream \
   --execution_mode=valid \
   --device=cuda \
   --quiet

# For Offline (Fails during Performance run)
mlcr run-mlperf,inference,_full,_r5.0-dev \
   --model=retinanet \
   --implementation=reference \
   --framework=onnxruntime \
   --category=edge \
   --scenario=Offline \
   --execution_mode=valid \
   --device=cuda \
   --quiet

# For MultiStream (Fails during Performance run)
mlcr run-mlperf,inference,_full,_r5.0-dev \
   --model=retinanet \
   --implementation=reference \
   --framework=onnxruntime \
   --category=edge \
   --scenario=MultiStream \
   --execution_mode=valid \
   --device=cuda \
   --quiet

Contrast: Successful ResNet50 Commands (Same Environment):

The following commands run successfully to completion (Performance and Accuracy) in the exact same environment:

# ResNet50 Edge Valid
mlcr run-mlperf,inference,_full,_r5.0-dev,_all-scenarios \
   --model=resnet50 \
   --implementation=reference \
   --framework=onnxruntime \
   --category=edge  \
   --execution_mode=valid \
   --device=cuda \
   --quiet

# ResNet50 Datacenter Valid (SERVER_TARGET_QPS needs to be set)
# mlcr run-mlperf,inference,_full,_r5.0-dev,_all-scenarios \
#    --model=resnet50 \
#    --implementation=reference \
#    --framework=onnxruntime \
#    --category=datacenter \
#    --server_target_qps=<SERVER_TARGET_QPS> \
#    --execution_mode=valid \
#    --device=cuda \
#    --quiet

Observed Behavior:

When running RetinaNet with the reference implementation and ONNX Runtime backend in valid mode, multiple distinct failures occur depending on the scenario:

1. SingleStream Scenario:

   • The Performance part runs successfully. Example summary output:

     ================================================
     MLPerf Results Summary
     ================================================
     SUT name : PySUT
     Scenario : SingleStream
     Mode     : PerformanceOnly
     90.0th percentile latency (ns) : 21396647
     Result is : VALID
     ... (rest of summary) ...
     ================================================
     

   • However, the subsequent Accuracy evaluation phase fails consistently with a ValueError. This happens even after clearing the results cache (rm -rf .../retinanet) and re-downloading/re-processing the dataset. The error occurs within pycocotools when processing the ground truth annotations.
   • Accuracy Traceback:

     loading annotations into memory...
     Done (t=0.46s)
     creating index...
     index created!
     Loading and preparing results...
     Converting ndarray to lists...
     (1773424, 7) # Note: Prediction count might vary slightly between runs
     0/1773424
     1000000/1773424
     DONE (t=10.84s)
     creating index...
     index created!
     Running per image evaluation...
     Evaluate annotation type *bbox*
     Traceback (most recent call last):
       File "/home/wanghaonan/MLC/repos/local/cache/get-git-repo_ecbc20bf/inference/vision/classification_and_detection/python/main.py", line 781, in <module>
         main()
       File "/home/wanghaonan/MLC/repos/local/cache/get-git-repo_ecbc20bf/inference/vision/classification_and_detection/python/main.py", line 757, in main
         post_proc.finalize(result_dict, ds, output_dir=args.output)
       File "/home/wanghaonan/MLC/repos/local/cache/get-git-repo_ecbc20bf/inference/vision/classification_and_detection/python/openimages.py", line 295, in finalize
         cocoEval.evaluate()
       File "/home/wanghaonan/miniconda3/envs/ml_perf/lib/python3.10/site-packages/pycocotools/cocoeval.py", line 148, in evaluate
         self.ious = {(imgId, catId): computeIoU(imgId, catId) \
       File "/home/wanghaonan/miniconda3/envs/ml_perf/lib/python3.10/site-packages/pycocotools/cocoeval.py", line 148, in <dictcomp>
         self.ious = {(imgId, catId): computeIoU(imgId, catId) \
       File "/home/wanghaonan/miniconda3/envs/ml_perf/lib/python3.10/site-packages/pycocotools/cocoeval.py", line 188, in computeIoU
         iscrowd = [int(o['iscrowd']) for o in gt]
       File "/home/wanghaonan/miniconda3/envs/ml_perf/lib/python3.10/site-packages/pycocotools/cocoeval.py", line 188, in <listcomp>
         iscrowd = [int(o['iscrowd']) for o in gt]
     ValueError: cannot convert float NaN to integer
     

   • This suggests the COCO annotation file generated by tools/openimages.py contains NaN values for the iscrowd field.

2. Offline Scenario:

   • The run fails during the Performance test itself, before even reaching accuracy evaluation.
   • It throws a CUDA error related to memory access.
   • CUDA Error Log Snippet:

     ERROR:main:thread: failed on contentid=['/home/wanghaonan/MLC/repos/local/cache/get-preprocessed-dataset-openimages_2952321f/7aa0b17d541db028.jpg'], [ONNXRuntimeError] : 1 : FAIL : CUDA failure 716: misaligned address ; GPU=0 ; hostname=graph-2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=63 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));
     2025-04-23 10:41:28.753762954 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 716: misaligned address ; GPU=0 ; hostname=graph-2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=63 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));
     2025-04-23 10:41:28.753789958 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 716: misaligned address ; GPU=0 ; hostname=graph-2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider.cc ; line=446 ; expr=cudaStreamSynchronize(static_cast<cudaStream_t>(stream_));
     

   • This CUDA error often coincides with Xid 31 errors reported in dmesg for the corresponding GPU and python3 process, indicating an MMU fault likely related to accessing an invalid address (e.g., near NULL).
   • Relevant dmesg Output:

     [10348879.702796] NVRM: Xid (PCI:0000:34:00): 31, pid=2992993, name=python3, Ch 00000008, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_6 faulted @ 0x0_00000000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ
     

     (Note: PID will change between runs, but the pattern of Xid 31 / MMU Fault / python3 / 0x0 address was observed)

3. MultiStream Scenario:

   • Similar to the Offline scenario, the run fails during the Performance test.
   • It exhibits the same CUDA failure 716: misaligned address error and corresponding Xid 31 errors in dmesg.
   • CUDA Error Log Snippet:

     ERROR:main:thread: failed on contentid=['/home/wanghaonan/MLC/repos/local/cache/get-preprocessed-dataset-openimages_2952321f/776b458d0d0e412a.jpg'], [ONNXRuntimeError] : 1 : FAIL : CUDA failure 716: misaligned address ; GPU=0 ; hostname=graph-2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=63 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));
     2025-04-23 11:13:47.424419492 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 716: misaligned address ; GPU=0 ; hostname=graph-2 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/gpu_data_transfer.cc ; line=63 ; expr=cudaMemcpyAsync(dst_data, src_data, bytes, cudaMemcpyHostToDevice, static_cast<cudaStream_t>(stream.GetHandle()));
     

Expected Behavior:

The mlcr command for RetinaNet using the reference implementation and ONNX Runtime in valid mode should successfully complete both Performance and Accuracy runs for all specified scenarios (SingleStream, MultiStream, Offline), similar to how ResNet50 runs successfully in the same environment. At a minimum, the Performance runs for MultiStream and Offline should not trigger CUDA errors or Xid events.

Attempts to Resolve:

• Cleared the relevant CM results cache (rm -rf /home/wanghaonan/MLC/repos/local/cache/get-mlperf-inference-results-dir_65ab07b1/valid_results/graph_2-reference-gpu-onnxruntime-v1.18.0-cu121/retinanet) multiple times.
• Allowed CM to re-download and re-process the OpenImages dataset.
• The ValueError in accuracy persists after data reprocessing.
• The CUDA errors in MultiStream/Offline persist after clearing results.

Conclusion:

Given that the environment successfully runs ResNet50 benchmarks, and the RetinaNet benchmark fails in multiple, specific ways (data generation issue for accuracy, CUDA memory issue for MultiStream/Offline) even after cache clearing, this strongly suggests potential bugs within the MLPerf reference scripts' handling of RetinaNet data preparation/batching for ONNX Runtime, or an issue in how ONNX Runtime's CUDA EP handles this specific model/data combination.

Could you please investigate these issues? Let me know if full logs for specific runs are needed.

Reproduce the RetinaNet ONNX Runtime commands in the issue, starting with inference/vision/classification_and_detection/python/main.py and openimages.py. Inspect the generated COCO annotations around the iscrowd field and compare the SingleStream, Offline, and MultiStream paths. Done means the reported accuracy and CUDA failures are resolved across the specified scenarios without NaN annotations or CUDA/Xid errors.