ONNX Runtime ResNet50 Inference on NVIDIA A30 Uses CPU Instead of GPU

Hi, everyone!

I was testing ResNet50 MLPerf benchmarking on Nvidia A30. I was following this docs: https://docs.mlcommons.org/inference/benchmarks/image_classification/resnet50/#__tabbed_32_1

When running the MLPerf Inference v5.1 benchmark, the inference appears to be running on the CPU instead of the GPU. Throughput is around ~160 samples/sec, significantly below expected GPU performance (~1000+).

Here's the related outputs and logs:

- No GPU utilization seen in nvidia-smi - htop confirms heavy CPU usage

This is final console output:
Graph visualization saved as /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_d5e35de0/valid_results/bf9334fda04a-reference-gpu-onnxruntime-v1.22.0-cu124/resnet50/offline/TEST04/mlc-deps.png
Mermaid syntax saved to /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_d5e35de0/valid_results/bf9334fda04a-reference-gpu-onnxruntime-v1.22.0-cu124/resnet50/offline/TEST04/mlc-deps.mmd
Graph data saved as graph.graphml
[2025-07-10 09:32:29,312 module.py:5274 INFO] - ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/draw-graph-from-json-data/customize.py
[2025-07-10 09:32:29,326 customize.py:339 INFO] - bf9334fda04a-reference-gpu-onnxruntime-v1.22.0-cu124
[2025-07-10 09:32:29,326 customize.py:342 INFO] - +----------+----------+----------+------------+-----------------+---------------------------------+--------+--------+
| Model | Scenario | Accuracy | Throughput | Latency (in ms) | Power Efficiency (in samples/J) | TEST01 | TEST04 |
+----------+----------+----------+------------+-----------------+---------------------------------+--------+--------+
| resnet50 | Offline | 76.456 | 162.262 | - | | passed | passed |
+----------+----------+----------+------------+-----------------+---------------------------------+--------+--------+

The MLPerf inference results are stored at /home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_d5e35de0/valid_results

[2025-07-10 09:32:29,327 module.py:5274 INFO] - ! call "postprocess" from /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py
[2025-07-10 09:32:29,330 customize.py:401 INFO] -
Path to the MLPerf inference benchmark reference sources: /home/mlcuser/MLC/repos/local/cache/get-git-repo_inference-src_48b4ca01/inference
Path to the MLPerf inference reference configuration file: /home/mlcuser/MLC/repos/local/cache/get-git-repo_inference-src_48b4ca01/inference/mlperf.conf
[2025-07-10 09:32:29,330 customize.py:406 INFO] -
root@bf9334fda04a:~#
As we can see from the console output both TEST01 and TEST04 are successfully passed.

Environment:

GPU: NVIDIA A30 (24GB)
Driver: 535.230.02
CUDA version: 12.2
ONNX Runtime version: 1.22.0
MLPerf Inference version: v5.1
MLCFlow / Docker: Yes (inside container)
Model: ResNet50 (ONNX)
Scenario: Offline
Batch Size: 32 (default)

Could you help identify the following:

1. Why is onnxruntime-gpu not utilizing the GPU during inference?
2. How can we ensure that the CUDAExecutionProvider is activated and used inside the Docker container?
3. Is the reference implementation of ONNX Runtime in MLPerf Inference designed to run only on the CPU, regardless of GPU availability?
4. Approximately, what throughput can be expected when running a ResNet ONNX model on an NVIDIA A30 GPU with a batch size of 32? Also, what is the default batch size?

I’d really appreciate any help or insights on this issue. Please let me know if more logs or environment details are needed.

Thanks in advance!

Start with the MLPerf Inference reference sources and inference/mlperf.conf named in the report, then inspect how the ONNX Runtime container selects execution providers. Reproduce the ResNet50 Offline run with the stated A30, Docker, and batch-size settings, checking CUDAExecutionProvider and GPU utilization. Done means the cause is identified and the benchmark uses the intended provider, with throughput and configuration documented.