Cross-compiling for linux/arm64 causes segmentation fault on hosts with Linux kernels newer than 6.8.0-49-generic

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Cross-compiling linux/arm64 Docker images using docker buildx results in segmentation faults during the build process on host systems running Linux kernel versions newer than 6.8.0-49-generic. Builds are successful on kernel 6.8.0-49-generic.

Expected behaviour

The docker buildx build process for linux/arm64 should complete successfully without segmentation faults on hosts with recent Linux kernels, including those newer than 6.8.0-49-generic.

Actual behaviour

The build process is interrupted by a segmentation fault when the host kernel is newer than 6.8.0-49-generic. The build completes successfully on kernel 6.8.0-49-generic.

Buildx version

github.com/docker/buildx v0.20.0 8e30c46

Docker info

Output from `docker version` (as `docker info` output was not directly provided):
Client: Docker Engine - Community
 Version:         27.5.1
 API version:     1.47
 Go version:      go1.22.11
 Git commit:      9f9e405
 Built:           Wed Jan 22 13:41:48 2025
 OS/Arch:         linux/amd64
 Context:         default

Server: Docker Engine - Community
 Engine:
  Version:         27.5.1
  API version:     1.47 (minimum version 1.24)
  Go version:      go1.22.11
  Git commit:      4c9b3b0
  Built:           Wed Jan 22 13:41:48 2025
  OS/Arch:         linux/amd64
  Experimental:    false
 containerd:
  Version:         1.7.25
  GitCommit:       bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
 runc:
  Version:         1.2.4
  GitCommit:       v1.2.4-0-g6c52b3f
 docker-init:
  Version:         0.19.0
  GitCommit:       de40ad0

Builders list

vpassamano@iitavr002lw013u:~/Desktop/ros1-ros2-zed-jetson-docker/ros1_noetic$ docker buildx ls
NAME/NODE        DRIVER/ENDPOINT                   STATUS    BUILDKIT   PLATFORMS
mybuilder*       docker-container                                       
 \_ mybuilder0    \_ unix:///var/run/docker.sock   running   v0.20.2    linux/amd64 (+4), linux/arm64, linux/arm (+2), linux/ppc64le, (4 more)
default          docker                                                 
 \_ default       \_ default                       running   v0.18.2    linux/amd64 (+4), linux/arm64, linux/arm (+2), linux/ppc64le, (7 more)

Configuration

Invoked Commands (general setup and build):

1. QEMU setup (run once on host): sudo docker run --rm --privileged multiarch/qemu-user-static --reset -p yes
2. Buildx builder creation (if not existing): docker buildx create --bootstrap --name mybuilder --driver docker-container --use
3. Build command (executed on a host with a problematic kernel newer than 6.8.0-49-generic):

   docker buildx build --builder mybuilder --platform linux/arm64 --tag my-arm64-image --load .
   

Dockerfile/Project:
A minimal Dockerfile to reliably reproduce the segmentation fault on problematic kernels is not yet available. The issue is observed when building complex ROS 2 Humble images for NVIDIA Jetson Orin (target: linux/arm64), which involve compiling numerous C++ packages (e.g., ZED SDK related components, PCL, OpenCV) and their dependencies. The segmentation fault occurs during RUN steps that execute compilation or other computationally intensive processes within the QEMU-emulated linux/arm64 environment.

Build logs

274.6 c++: internal compiler error: Segmentation fault signal terminated program cc1plus
...
274.6 make[2]: *** [rospack/CMakeFiles/rospackexe.dir/build.make:76: rospack/CMakeFiles/rospackexe.dir/src/rospack_main.cpp.o] Error 4

Additional info

Host System Details (where kernel 6.8.0-49-generic works, and newer ones fail):

• Host OS: Ubuntu 24.04.1 LTS (Codename: noble)
• Host Kernel (Last Known Good): Linux iitavr002lw013u 6.8.0-49-generic #49-Ubuntu SMP PREEMPT_DYNAMIC Mon Nov 4 02:06:24 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux (Builds complete successfully with this kernel).
• Host Kernel (Problematic): kernel_version[>=6.8.0-51-generic]
• QEMU Package Version: qemu-user-static, Version: 1:8.2.2+ds-0ubuntu1.6
• QEMU Binary Version (aarch64): qemu-aarch64 version 8.2.2 (Debian 1:8.2.2+ds-0ubuntu1.6)

Host Cross-Compilation Toolchain (present on the system, from the setup with the working kernel):

• aarch64-linux-gnu-gcc (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
• aarch64-linux-gnu-g++ (Ubuntu 13.3.0-6ubuntu2~24.04) 13.3.0
• libc6-dev-arm64-cross: 2.39-0ubuntu8cross1
• libstdc++-13-dev-arm64-cross: 13.3.0-6ubuntu2~24.04cross1

Hypothesis:
The issue might stem from an incompatibility or interaction between newer Linux host kernels (those tested being > 6.8.0-49-generic), the binfmt_misc mechanism, and QEMU's user-mode emulation for aarch64. docker buildx relies on this underlying stack for cross-platform builds. Since Buildx is the primary user-facing tool orchestrating this workflow, this issue is being reported here for triage and potential investigation into the interaction.

Any insights, potential workarounds (other than downgrading the host kernel, if possible), or guidance on further debugging this segmentation fault during cross-compilation would be greatly appreciated.

Start by reproducing the failure with the documented QEMU setup, the mybuilder builder, and docker buildx build --platform linux/arm64, comparing kernels 6.8.0-49-generic and newer versions. Use the reported cc1plus segmentation fault and the binfmt_misc/QEMU hypothesis to narrow the failing layer; a minimal reproducible Dockerfile and a confirmed fix or documented workaround would define completion.