Resnet50 MLPerf 4.1 Attribute Error: ResNet50EngineBuilder object has no attribute calibrator

We are a test team her trying to run MLPerf on a server with Nvidia GPUs / Cuda for a certification run. We are using the following script on Rocky Linux:

Into container

cd ~/NVIDIA

make prebuild

Enviroment setting in container

export USER=$(whoami)
sudo usermod -aG root $(whoami)
sudo chown -R $(whoami):$(whoami) /work
sudo chown -R $(whoami):$(whoami) ~/scratch

Install Python packages in container

pip install opencv-python-headless mlperf_logging simplejson

Download model

make download_model BENCHMARKS="resnet50"

Download Dataset

make download_data BENCHMARKS="resnet50"
make preprocess_data BENCHMARKS="resnet50"

Run Benchmark

make run RUN_ARGS="--benchmarks=resnet50 --scenarios=offline,server"

The run gives the error mentioned in the headline.

Resnet50 MLPerf 4.1 Attribute Error: ResNet50EngineBuilder object has no attribute calibrator

Any help appreciated, thank you.

---------------- added: From the terminal: ----------------

(mlperf) s370@mlperf-inference-s370-x86-64-21773:/work$ make run RUN_ARGS="--benchmarks=resnet50 --scenarios=offline,server"
make[1]: Entering directory '/work'
[2025-02-21 12:33:17,741 main.py:229 INFO] Detected system ID: KnownSystem.S370
[2025-02-21 12:33:25,598 generate_engines.py:171 INFO] Building engines for resnet50 benchmark in Offline scenario...
[2025-02-21 12:33:25,602 builder.py:460 WARNING] No batch_size dict provided for ResNet50EngineBuilderOp. Setting to default value {<ResNet50Component.ResNet50: AliasedName(name='resnet50', aliases=('resnet',), patterns=())>: 1}
[02/21/2025-12:33:25] [TRT] [I] [MemUsageChange] Init CUDA: CPU +22, GPU +0, now: CPU 40, GPU 446 (MiB)
[02/21/2025-12:33:27] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +2170, GPU +414, now: CPU 2363, GPU 860 (MiB)
Process Process-1:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/code/actionhandler/base.py", line 189, in subprocess_target
return self.action_handler.handle()
File "/work/code/actionhandler/generate_engines.py", line 174, in handle
total_engine_build_time += self.build_engine(job)
File "/work/code/actionhandler/generate_engines.py", line 165, in build_engine
builder.build_engines()
File "/usr/local/lib/python3.10/dist-packages/nvmitten/nvidia/builder.py", line 570, in build_engines
self.mitten_builder.run(self.legacy_scratch, None)
File "/usr/local/lib/python3.10/dist-packages/nvmitten/debug/debug_manager.py", line 256, in _wrapper
raise exc_info[1]
File "/usr/local/lib/python3.10/dist-packages/nvmitten/debug/debug_manager.py", line 243, in _wrapper
retval = obj(*args, **kwargs)
File "/work/code/resnet50/tensorrt/builder.py", line 472, in run
builder_config = builder.create_builder_config()
File "/work/code/resnet50/tensorrt/builder.py", line 258, in create_builder_config
builder_config.int8_calibrator = self.calibrator
AttributeError: 'ResNet50EngineBuilder' object has no attribute 'calibrator'
[2025-02-21 12:33:31,286 generate_engines.py:171 INFO] Building engines for resnet50 benchmark in Offline scenario...
[2025-02-21 12:33:31,287 builder.py:460 WARNING] No batch_size dict provided for ResNet50EngineBuilderOp. Setting to default value {<ResNet50Component.ResNet50: AliasedName(name='resnet50', aliases=('resnet',), patterns=())>: 1}
[02/21/2025-12:33:31] [TRT] [I] [MemUsageChange] Init CUDA: CPU +21, GPU +0, now: CPU 40, GPU 446 (MiB)
[02/21/2025-12:33:33] [TRT] [I] [MemUsageChange] Init builder kernel library: CPU +2170, GPU +414, now: CPU 2363, GPU 860 (MiB)
Process Process-2:
Traceback (most recent call last):
File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/work/code/actionhandler/base.py", line 189, in subprocess_target
return self.action_handler.handle()
File "/work/code/actionhandler/generate_engines.py", line 174, in handle
total_engine_build_time += self.build_engine(job)
File "/work/code/actionhandler/generate_engines.py", line 165, in build_engine
builder.build_engines()
File "/usr/local/lib/python3.10/dist-packages/nvmitten/nvidia/builder.py", line 570, in build_engines
self.mitten_builder.run(self.legacy_scratch, None)
File "/usr/local/lib/python3.10/dist-packages/nvmitten/debug/debug_manager.py", line 256, in _wrapper
raise exc_info[1]
File "/usr/local/lib/python3.10/dist-packages/nvmitten/debug/debug_manager.py", line 243, in _wrapper
retval = obj(*args, **kwargs)
File "/work/code/resnet50/tensorrt/builder.py", line 472, in run
builder_config = builder.create_builder_config()
File "/work/code/resnet50/tensorrt/builder.py", line 258, in create_builder_config
builder_config.int8_calibrator = self.calibrator
AttributeError: 'ResNet50EngineBuilder' object has no attribute 'calibrator'
Traceback (most recent call last):
File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/work/code/main.py", line 231, in
main(main_args, DETECTED_SYSTEM)
File "/work/code/main.py", line 144, in main
dispatch_action(main_args, config_dict, workload_setting)
File "/work/code/main.py", line 202, in dispatch_action
handler.run()
File "/work/code/actionhandler/base.py", line 82, in run
self.handle_failure()
File "/work/code/actionhandler/base.py", line 186, in handle_failure
self.action_handler.handle_failure()
File "/work/code/actionhandler/generate_engines.py", line 182, in handle_failure
raise RuntimeError("Building engines failed!")
RuntimeError: Building engines failed!
make[1]: *** [Makefile:37: generate_engines] Error 1
make[1]: Leaving directory '/work'
make: *** [Makefile:31: run] Error 2

Start with code/resnet50/tensorrt/builder.py, especially create_builder_config at the traceback's reported line, and trace how ResNet50EngineBuilder is initialized before builder_config.int8_calibrator is assigned. Reproduce with make run RUN_ARGS="--benchmarks=resnet50 --scenarios=offline,server" and confirm that engine generation completes without the reported AttributeError.