microsoft / microsoft/onnxruntime
[Build] Docker build failure with ROCm 6.0 using official Dockerfile for v1.19.2: Segmentation fault in clang++ during composable_kernel compilation
- Dominant language
- C++
- Stars
- 21.9k
- Forks
- 4.2k
- Avg merge
- 4d 11h
- Merged PRs (30d)
- 184
Description
### Describe the issue
Building the ONNX Runtime Docker image using the official Dockerfile.rocm for version v1.19.2 with ROCm 6.0 fails with a segmentation fault during the CMake build process. The error occurs when compiling a file related to composable_kernel and FMHA (fmha_fwd_d64_fp8_batch_b128x64x32x64x32x64_r2x1x1_w32x32x32_qr_vc_squant.cpp.o).
This issue prevents the successful building of the ROCm Docker image for ONNX Runtime v1.19.2 using the official Dockerfile.
**Environment**:
Base Docker Image: rocm/pytorch:rocm6.0_ubuntu20.04_py3.9_pytorch_2.1.1
ONNX Runtime Version: v1.19.2
ROCm Version: 6.0.0
Dockerfile: Dockerfile.rocm (content provided below)
Build Log: build.log.txt (attached below)
**Steps to reproduce:**
1. Create a file named Dockerfile.rocm with the following content:
```
# --------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
# --------------------------------------------------------------
# Dockerfile to run ONNXRuntime with ROCm integration
#--------------------------------------------------------------------------
FROM rocm/pytorch:rocm6.0_ubuntu20.04_py3.9_pytorch_2.1.1
ARG ONNXRUNTIME_REPO=https://github.com/Microsoft/onnxruntime
ARG ONNXRUNTIME_BRANCH=v1.19.2
WORKDIR /code
ENV PATH /code/cmake-3.27.3-linux-x86_64/bin:${PATH}
# Prepare onnxruntime repository & build onnxruntime
RUN git clone --single-branch --branch ${ONNXRUNTIME_BRANCH} --recursive ${ONNXRUNTIME_REPO} onnxruntime &&\
/bin/sh onnxruntime/dockerfiles/scripts/install_common_deps.sh &&\
cd onnxruntime &&\
/bin/sh ./build.sh --allow_running_as_root --config Release --build_wheel --update --build --parallel --cmake_extra_defines\
ONNXRUNTIME_VERSION=$(cat ./VERSION_NUMBER) --use_rocm --rocm_home=/opt/rocm &&\
pip install /code/onnxruntime/build/Linux/Release/dist/*.whl &&\
cd ..
```
2. Build the docker image using:
`docker build -t onnxruntime_rocm -f Dockerfile.rocm .`
**Error Log Snippet:**
```
clang++: error: unable to execute command: Segmentation fault (core dumped)
clang++: error: clang frontend command failed due to signal (use -v to see invocation)
AMD clang version 17.0.0
...
subprocess.CalledProcessError: Command '['/code/cmake-3.27.3-linux-x86_64/bin/cmake', '--build', '/code/onnxruntime/build/Linux/Release', '--config', 'Release', '--', '-j24']' returned non-zero exit status 2.
```
### Urgency
_No response_
### Target platform
x86-64
### Build script
```
# --------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
# --------------------------------------------------------------
# Dockerfile to run ONNXRuntime with ROCm integration
#--------------------------------------------------------------------------
FROM rocm/pytorch:rocm6.0_ubuntu20.04_py3.9_pytorch_2.1.1
ARG ONNXRUNTIME_REPO=https://github.com/Microsoft/onnxruntime
ARG ONNXRUNTIME_BRANCH=v1.19.2
WORKDIR /code
ENV PATH /code/cmake-3.27.3-linux-x86_64/bin:${PATH}
# Prepare onnxruntime repository & build onnxruntime
RUN git clone --single-branch --branch ${ONNXRUNTIME_BRANCH} --recursive ${ONNXRUNTIME_REPO} onnxruntime &&\
/bin/sh onnxruntime/dockerfiles/scripts/install_common_deps.sh &&\
cd onnxruntime &&\
/bin/sh ./build.sh --allow_running_as_root --config Release --build_wheel --update --build --parallel --cmake_extra_defines\
ONNXRUNTIME_VERSION=$(cat ./VERSION_NUMBER) --use_rocm --rocm_home=/opt/rocm &&\
pip install /code/onnxruntime/build/Linux/Release/dist/*.whl &&\
cd ..
```
### Error / output
[build.log.txt](https://github.com/user-attachments/files/18952883/build.log.txt)
### Visual Studio Version
_No response_
### GCC / Compiler Version
_No response_
Contributor guide
Research direction
Start with Dockerfile.rocm and reproduce the v1.19.2 build using the documented docker build command. Read build.log.txt around the composable_kernel FMHA object and the clang++ segmentation fault, then verify the Docker image builds successfully with the provided ROCm and PyTorch base image.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cmake, cpp, docker, pytorch
- Domain
- build-system, devops, machine-learning
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100