mlcommons / mlcommons/inference
Fail to build the docker for mlc command on Ubuntu 22.04
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 1.6k
- Forks
- 650
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 6
Description
I failed to run the mlc command on Ubuntu 22.04:
mlcr run-mlperf,inference,_find-performance,_full,_r4.1-dev
--model=resnet50
--implementation=nvidia
--framework=tensorrt
--category=edge
--scenario=Offline
--execution_mode=test
--device=cuda
--docker --quiet
--test_query_count=5000
--all_models=yes
Failed to resolve 'developer.download.nvidia.com' as below, but in fact I can access the link of developer.download.nvidia.com via firefox manually.
212.3 The following NEW packages will be installed:
212.3 libcublas-12-3 libcublas-dev-12-3
242.8 0 upgraded, 2 newly installed, 0 to remove and 87 not upgraded.
242.8 Need to get 514 MB of archives.
242.8 After this operation, 1577 MB of additional disk space will be used.
242.8 Ign:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 libcublas-12-3 12.3.4.1-1
262.8 Ign:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 libcublas-dev-12-3 12.3.4.1-1
282.9 Ign:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 libcublas-12-3 12.3.4.1-1
282.9 Ign:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 libcublas-dev-12-3 12.3.4.1-1
284.9 Ign:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 libcublas-12-3 12.3.4.1-1
284.9 Ign:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 libcublas-dev-12-3 12.3.4.1-1
308.9 Err:1 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 libcublas-12-3 12.3.4.1-1
308.9 Could not connect to developer.download.nvidia.com:443 (23.223.211.90), connection timed out Could not connect to developer.download.nvidia.com:443 (23.223.211.42), connection timed out
328.9 Err:2 https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64 libcublas-dev-12-3 12.3.4.1-1
328.9 Temporary failure resolving 'developer.download.nvidia.com'
328.9 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/./libcublas-12-3_12.3.4.1-1_amd64.deb Could not connect to developer.download.nvidia.com:443 (23.223.211.90), connection timed out Could not connect to developer.download.nvidia.com:443 (23.223.211.42), connection timed out
328.9 E: Failed to fetch https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/./libcublas-dev-12-3_12.3.4.1-1_amd64.deb Temporary failure resolving 'developer.download.nvidia.com'
328.9 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
------
4 warnings found (use docker --debug to expand):
- FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 6)
- FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 14)
- FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 53)
- FromAsCasing: 'as' and 'FROM' keywords' casing do not match (line 66)
Dockerfile.multi:32
--------------------
31 | COPY docker/common/install_tensorrt.sh install_tensorrt.sh
32 | >>> RUN bash ./install_tensorrt.sh \
33 | >>> --TRT_VER=${TRT_VER} \
34 | >>> --CUDA_VER=${CUDA_VER} \
35 | >>> --CUDNN_VER=${CUDNN_VER} \
36 | >>> --NCCL_VER=${NCCL_VER} \
37 | >>> --CUBLAS_VER=${CUBLAS_VER} && \
38 | >>> rm install_tensorrt.sh
39 |
--------------------
ERROR: failed to solve: process "/bin/bash -c bash ./install_tensorrt.sh --TRT_VER=${TRT_VER} --CUDA_VER=${CUDA_VER} --CUDNN_VER=${CUDNN_VER} --NCCL_VER=${NCCL_VER} --CUBLAS_VER=${CUBLAS_VER} && rm install_tensorrt.sh" did not complete successfully: exit code: 100
exit status 1
make: *** [Makefile:55: devel_build] Error 1
make: Leaving directory '/home/bob2/MLC/repos/local/cache/get-git-repo_d790359e/repo/docker'
Traceback (most recent call last):
File "/home/bob2/mlc/bin/mlcr", line 8, in <module>
sys.exit(mlcr())
File "/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py", line 1715, in mlcr
main()
File "/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py", line 1797, in main
res = method(run_args)
File "/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py", line 1529, in run
return self.call_script_module_function("run", run_args)
File "/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py", line 1509, in call_script_module_function
result = automation_instance.run(run_args) # Pass args to the run method
File "/home/bob2/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 225, in run
r = self._run(i)
File "/home/bob2/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 1768, in _run
r = customize_code.preprocess(ii)
File "/home/bob2/MLC/repos/mlcommons@mlperf-automations/script/run-mlperf-inference-app/customize.py", line 284, in preprocess
r = mlc.access(ii)
File "/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py", line 92, in access
result = method(options)
File "/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py", line 1526, in docker
return self.call_script_module_function("docker", run_args)
File "/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py", line 1511, in call_script_module_function
result = automation_instance.docker(run_args) # Pass args to the run method
File "/home/bob2/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 4684, in docker
return docker_run(self, i)
File "/home/bob2/MLC/repos/mlcommons@mlperf-automations/automation/script/docker.py", line 308, in docker_run
r = self_module._run_deps(
File "/home/bob2/MLC/repos/mlcommons@mlperf-automations/automation/script/module.py", line 3695, in _run_deps
r = self.action_object.access(ii)
File "/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py", line 92, in access
result = method(options)
File "/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py", line 1529, in run
return self.call_script_module_function("run", run_args)
File "/home/bob2/mlc/lib/python3.10/site-packages/mlc/main.py", line 1519, in call_script_module_function
raise ScriptExecutionError(f"Script {function_name} execution failed. Error : {error}")
mlc.main.ScriptExecutionError: Script run execution failed. Error : MLC script failed (name = get-ml-model-gptj, return code = 256)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Please file an issue at https://github.com/mlcommons/mlperf-automations/issues along with the full MLC command being run and the relevant
or full console log.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with Dockerfile.multi:32 and docker/common/install_tensorrt.sh, then reproduce the devel_build target from the Docker directory using the reported Ubuntu 22.04 command. Determine why the CUDA package downloads fail during the TensorRT installation and verify that the Docker build completes successfully or that the failure is clearly documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, python, ubuntu
- Domain
- build-system, devops, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100