microsoft / microsoft/onnxruntime

[Build] Docker build failure with ROCm 6.0 using official Dockerfile for v1.19.2: Segmentation fault in clang++ during composable_kernel compilation

Open
#23,807 1 comment 0 reactions 0 assignees View on GitHub
build contributions welcome
Dominant language
C++
Stars
21.9k
Forks
4.2k
Avg merge
4d 11h
Merged PRs (30d)
184

Description

### Describe the issue

Building the ONNX Runtime Docker image using the official Dockerfile.rocm for version v1.19.2 with ROCm 6.0 fails with a segmentation fault during the CMake build process. The error occurs when compiling a file related to composable_kernel and FMHA (fmha_fwd_d64_fp8_batch_b128x64x32x64x32x64_r2x1x1_w32x32x32_qr_vc_squant.cpp.o).

This issue prevents the successful building of the ROCm Docker image for ONNX Runtime v1.19.2 using the official Dockerfile.

**Environment**:

Base Docker Image: rocm/pytorch:rocm6.0_ubuntu20.04_py3.9_pytorch_2.1.1
ONNX Runtime Version: v1.19.2
ROCm Version: 6.0.0
Dockerfile: Dockerfile.rocm (content provided below)
Build Log: build.log.txt (attached below)

**Steps to reproduce:**

1. Create a file named Dockerfile.rocm with the following content:

```
# --------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
# --------------------------------------------------------------
# Dockerfile to run ONNXRuntime with ROCm integration
#--------------------------------------------------------------------------

FROM rocm/pytorch:rocm6.0_ubuntu20.04_py3.9_pytorch_2.1.1

ARG ONNXRUNTIME_REPO=https://github.com/Microsoft/onnxruntime
ARG ONNXRUNTIME_BRANCH=v1.19.2

WORKDIR /code

ENV PATH /code/cmake-3.27.3-linux-x86_64/bin:${PATH}

# Prepare onnxruntime repository & build onnxruntime
RUN git clone --single-branch --branch ${ONNXRUNTIME_BRANCH} --recursive ${ONNXRUNTIME_REPO} onnxruntime &&\
/bin/sh onnxruntime/dockerfiles/scripts/install_common_deps.sh &&\
cd onnxruntime &&\
/bin/sh ./build.sh --allow_running_as_root --config Release --build_wheel --update --build --parallel --cmake_extra_defines\
ONNXRUNTIME_VERSION=$(cat ./VERSION_NUMBER) --use_rocm --rocm_home=/opt/rocm &&\
pip install /code/onnxruntime/build/Linux/Release/dist/*.whl &&\
cd ..
```

2. Build the docker image using:

`docker build -t onnxruntime_rocm -f Dockerfile.rocm .`

**Error Log Snippet:**

```
clang++: error: unable to execute command: Segmentation fault (core dumped)
clang++: error: clang frontend command failed due to signal (use -v to see invocation)
AMD clang version 17.0.0
...
subprocess.CalledProcessError: Command '['/code/cmake-3.27.3-linux-x86_64/bin/cmake', '--build', '/code/onnxruntime/build/Linux/Release', '--config', 'Release', '--', '-j24']' returned non-zero exit status 2.
```

### Urgency

_No response_

### Target platform

x86-64

### Build script

```
# --------------------------------------------------------------
# Copyright (c) Microsoft Corporation. All rights reserved.
# Licensed under the MIT License.
# --------------------------------------------------------------
# Dockerfile to run ONNXRuntime with ROCm integration
#--------------------------------------------------------------------------

FROM rocm/pytorch:rocm6.0_ubuntu20.04_py3.9_pytorch_2.1.1

ARG ONNXRUNTIME_REPO=https://github.com/Microsoft/onnxruntime
ARG ONNXRUNTIME_BRANCH=v1.19.2

WORKDIR /code

ENV PATH /code/cmake-3.27.3-linux-x86_64/bin:${PATH}

# Prepare onnxruntime repository & build onnxruntime
RUN git clone --single-branch --branch ${ONNXRUNTIME_BRANCH} --recursive ${ONNXRUNTIME_REPO} onnxruntime &&\
/bin/sh onnxruntime/dockerfiles/scripts/install_common_deps.sh &&\
cd onnxruntime &&\
/bin/sh ./build.sh --allow_running_as_root --config Release --build_wheel --update --build --parallel --cmake_extra_defines\
ONNXRUNTIME_VERSION=$(cat ./VERSION_NUMBER) --use_rocm --rocm_home=/opt/rocm &&\
pip install /code/onnxruntime/build/Linux/Release/dist/*.whl &&\
cd ..
```

### Error / output

[build.log.txt](https://github.com/user-attachments/files/18952883/build.log.txt)

### Visual Studio Version

_No response_

### GCC / Compiler Version

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with Dockerfile.rocm and reproduce the v1.19.2 build using the documented docker build command. Read build.log.txt around the composable_kernel FMHA object and the clang++ segmentation fault, then verify the Docker image builds successfully with the provided ROCm and PyTorch base image.

Written by the indexing model from the issue text.

Assessment

Tech stack
cmake, cpp, docker, pytorch
Domain
build-system, devops, machine-learning
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.