[BUG] ResNet50 TensorRT build hangs at 92% in NMSOptPlugin during MLCFlow Inference (CUDA 12.2 + A30)

While attempting to benchmark ResNet50 under the Datacenter / Offline scenario using MLCFlow with NVIDIA implementation (TensorRT), the build hangs indefinitely during the compilation of the NMSOptPlugin.
This happens at [ 92%] with no CPU or GPU activity. It seems to hang at or right after building softmaxScore.cpp.o. No error is printed, only warnings.

OS: Ubuntu 22.04

I was following this docs:
https://docs.mlcommons.org/inference/benchmarks/image_classification/resnet50/#__tabbed_1_2

What I've done so far:

python3 -m venv mlc
source mlc/bin/activate
pip install mlcflow && mlc pull repo --url=mlcommons@cm4mlops --branch=mlperf-inference
mlc pull repo mlcommons@mlperf-automations --branch=dev
mlcr run-mlperf,inference,_find-performance,_full,_r5.0-dev
--model=resnet50
--implementation=nvidia
--framework=tensorrt
--category=datacenter
--scenario=Offline
--execution_mode=test
--device=cuda
--gpu_name=a30
--docker
--docker_cache=no
--quiet
--test_query_count=5000
--rerun
This is the freezing phase, I've waited for over an hour:
/home/mlcuser/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_851d6966/repo/closed/NVIDIA/code/plugin/NMSOptPlugin/src/nmsPluginOpt.h(136): warning https://github.com/mlcommons/inference/pull/997: function "nvinfer1::IPluginV2Ext::configurePlugin(const nvinfer1::Dims *, int32_t, const nvinfer1::Dims *, int32_t, const nvinfer1::DataType *, const nvinfer1::DataType *, const __nv_bool , const __nv_bool , nvinfer1::PluginFormat, int32_t)" is hidden by "nvinfer1::plugin::DetectionOutputOptDynamic::configurePlugin" -- virtual function override intended?
void configurePlugin(const DynamicPluginTensorDesc in, int nbInputs, const DynamicPluginTensorDesc out,
^

/home/mlcuser/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_851d6966/repo/closed/NVIDIA/code/plugin/NMSOptPlugin/src/nmsPluginOpt.h:101:46: warning: ‘IPluginV2Ext’ is deprecated [-Wdeprecated-declarations]
101 | IPluginV2Ext* clone() const noexcept override;
| ^~~~~
/usr/include/x86_64-linux-gnu/NvInferRuntimePlugin.h:447:22: note: declared here
447 | class TRT_DEPRECATED IPluginV2Ext : public IPluginV2
| ^~~~~~~~~~~~
/home/mlcuser/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_851d6966/repo/closed/NVIDIA/code/plugin/NMSOptPlugin/src/nmsPluginOpt.h:158:95: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations]
158 | IPluginV2* createPlugin(const char* name, const PluginFieldCollection* fc) noexcept override;
| ^~~
/usr/include/x86_64-linux-gnu/NvInferRuntimePlugin.h:117:22: note: declared here
117 | class TRT_DEPRECATED IPluginV2
| ^~~~~~~~~
/home/mlcuser/MLC/repos/local/cache/get-git-repo_mlperf-inferenc_851d6966/repo/closed/NVIDIA/code/plugin/NMSOptPlugin/src/nmsPluginOpt.h:160:112: warning: ‘IPluginV2’ is deprecated [-Wdeprecated-declarations]
160 | IPluginV2* deserializePlugin(const char* name, const void* serialData, size_t serialLength) noexcept override;
| ^~~
/usr/include/x86_64-linux-gnu/NvInferRuntimePlugin.h:117:22: note: declared here
117 | class TRT_DEPRECATED IPluginV2
| ^~~~~~~~~

Forcefully stopped:
Traceback (most recent call last):
File "/usr/local/bin/mlcr", line 8, in
sys.exit(mlcr())
File "/usr/local/lib/python3.8/dist-packages/mlc/main.py", line 86, in mlcr
main()
File "/usr/local/lib/python3.8/dist-packages/mlc/main.py", line 273, in main
res = method(run_args)
File "/usr/local/lib/python3.8/dist-packages/mlc/script_action.py", line 309, in run
return self.call_script_module_function("run", run_args)
File "/usr/local/lib/python3.8/dist-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 227, in run
r = self._run(i)
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1858, in _run
r = self._call_run_deps(prehook_deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3322, in _call_run_deps
r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3495, in _run_deps
r = self.action_object.access(ii)
File "/usr/local/lib/python3.8/dist-packages/mlc/action.py", line 56, in access
result = method(options)
File "/usr/local/lib/python3.8/dist-packages/mlc/script_action.py", line 309, in run
return self.call_script_module_function("run", run_args)
File "/usr/local/lib/python3.8/dist-packages/mlc/script_action.py", line 230, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 227, in run
r = self._run(i)
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1643, in _run
r = self._call_run_deps(deps, self.local_env_keys, local_env_keys_from_meta, env, state, const, const_state, add_deps_recursive,
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3322, in _call_run_deps
r = script._run_deps(deps, local_env_keys, env, state, const, const_state, add_deps_recursive, recursion_spaces,
File "/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3495, in _run_deps
r = self.action_object.access(ii)
File "/usr/local/lib/python3.8/dist-packages/mlc/action.py", line 56, in access
result = method(options)
File "/usr/local/lib/python3.8/dist-packages/mlc/script_action.py", line 309, in run
return self.call_script_module_function("run", run_args)
File "/usr/local/lib/python3.8/dist-packages/mlc/script_action.py", line 244, in call_script_module_function
raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.script_action.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = build-mlperf-inference-server-nvidia, return code = 2)

^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.

Expected Behavior:
MLCFlow should finish building the plugin and proceed to inference.

I’d really appreciate any help or insights on this issue. Please let me know if more logs or environment details are needed.

Thanks in advance!

Reproduce the ResNet50 Datacenter/Offline command and inspect the build output around NMSOptPlugin/src/nmsPluginOpt.h and softmaxScore.cpp.o. Compare the warnings and final return code with the expected plugin build flow; done means the NMSOptPlugin build completes and inference proceeds.