mlcommons / mlcommons/inference
SingleStream scenario crashes for Resnet50
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 650
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 6
Description
Run command used :
mlcr run-mlperf,inference,_r5.0-dev \
--model=resnet50 \
--implementation=reference \
--framework=onnxruntime \
--category=edge \
--scenario=SingleStream \
--execution_mode=valid \
--device=cpu \
--quiet
Crash log:
./run_local.sh onnxruntime resnet50 cpu --scenario SingleStream --threads 4 --user_conf '/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/33db40839ea34f5a94ac82e56583382a.conf' --use_preprocessed_dataset --cache_dir /home/mlcuser/MLC/repos/local/cache/get-preprocessed-dataset-imagenet_be6302c3 --dataset-list /home/mlcuser/MLC/repos/local/cache/extract-file_1b015636/val.txt 2>&1 | tee '/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_2bad8c5e/valid_results/6dc4971019f4-reference-cpu-onnxruntime-v1.20.1-default_config/resnet50/singlestream/performance/run_1/console.out'; echo ${PIPESTATUS[0]} > exitstatus
python3 python/main.py --profile resnet50-onnxruntime --model "/home/mlcuser/MLC/repos/local/cache/download-file_90fb1a82/resnet50_v1.onnx" --dataset-path /home/mlcuser/MLC/repos/local/cache/get-preprocessed-dataset-imagenet_be6302c3 --output "/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_2bad8c5e/valid_results/6dc4971019f4-reference-cpu-onnxruntime-v1.20.1-default_config/resnet50/singlestream/performance/run_1" --scenario SingleStream --threads 4 --user_conf /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/33db40839ea34f5a94ac82e56583382a.conf --use_preprocessed_dataset --cache_dir /home/mlcuser/MLC/repos/local/cache/get-preprocessed-dataset-imagenet_be6302c3 --dataset-list /home/mlcuser/MLC/repos/local/cache/extract-file_1b015636/val.txt
INFO:main:Namespace(dataset='imagenet', dataset_path='/home/mlcuser/MLC/repos/local/cache/get-preprocessed-dataset-imagenet_be6302c3', dataset_list='/home/mlcuser/MLC/repos/local/cache/extract-file_1b015636/val.txt', data_format=None, profile='resnet50-onnxruntime', scenario='SingleStream', max_batchsize=32, model='/home/mlcuser/MLC/repos/local/cache/download-file_90fb1a82/resnet50_v1.onnx', output='/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_2bad8c5e/valid_results/6dc4971019f4-reference-cpu-onnxruntime-v1.20.1-default_config/resnet50/singlestream/performance/run_1', inputs=None, outputs=['ArgMax:0'], backend='onnxruntime', device=None, model_name='resnet50', threads=4, qps=None, cache=0, cache_dir='/home/mlcuser/MLC/repos/local/cache/get-preprocessed-dataset-imagenet_be6302c3', preprocessed_dir=None, use_preprocessed_dataset=True, accuracy=False, find_peak_performance=False, debug=False, user_conf='/home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/33db40839ea34f5a94ac82e56583382a.conf', audit_conf='audit.config', time=None, count=None, performance_sample_count=None, max_latency=None, samples_per_query=8)
INFO:imagenet:Loading 50000 preprocessed images using 4 threads
INFO:imagenet:loaded 50000 images, cache=0, already_preprocessed=True, took=1.7sec
INFO:main:starting TestScenario.SingleStream
./run_local.sh: line 30: 4224 Killed python3 python/main.py --profile resnet50-onnxruntime --model "/home/mlcuser/MLC/repos/local/cache/download-file_90fb1a82/resnet50_v1.onnx" --dataset-path /home/mlcuser/MLC/repos/local/cache/get-preprocessed-dataset-imagenet_be6302c3 --output "/home/mlcuser/MLC/repos/local/cache/get-mlperf-inference-results-dir_2bad8c5e/valid_results/6dc4971019f4-reference-cpu-onnxruntime-v1.20.1-default_config/resnet50/singlestream/performance/run_1" --scenario SingleStream --threads 4 --user_conf /home/mlcuser/MLC/repos/mlcommons@mlperf-automations/script/generate-mlperf-inference-user-conf/tmp/33db40839ea34f5a94ac82e56583382a.conf --use_preprocessed_dataset --cache_dir /home/mlcuser/MLC/repos/local/cache/get-preprocessed-dataset-imagenet_be6302c3 --dataset-list /home/mlcuser/MLC/repos/local/cache/extract-file_1b015636/val.txt
Traceback (most recent call last):
File "/home/mlcuser/venv/mlc/bin/mlcr", line 8, in
sys.exit(mlcr())
File "/home/mlcuser/venv/mlc/lib/python3.10/site-packages/mlc/main.py", line 1354, in mlcr
main()
File "/home/mlcuser/venv/mlc/lib/python3.10/site-packages/mlc/main.py", line 1441, in main
res = method(run_args)
...
Screenshot from htop and System monitor during the execution:
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the command using run_local.sh and python/main.py with the resnet50-onnxruntime profile and SingleStream scenario. Inspect memory use around loading the 50,000 preprocessed images and starting TestScenario.SingleStream; done means the reported run completes without being killed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100