moby / moby/buildkit

QEMU emulated OpenTofu sporadically crashes but only if a builder is used

Open
#6,445 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/qemu kind/bug
Dominant language
Go
Stars
10.3k
Forks
1.5k
Avg merge
1d 23h
Merged PRs (30d)
48

Description

Description

OpenTofu will sporadically crash when running under emulation (e.g. while doing a arm64 docker build where the host platform is x86_64/amd64) but ONLY when a (custom) builder is being used.

The crashes will look like the following:

[...]
0.050 + cd /terraform                                                                                                            
0.051 + tofu providers lock
0.557 runtime: marked free object in span 0x2aaaf213f848, elemsize=384 freeindex=3 (bad use of unsafe.Pointer or having race conditions? try -d=checkptr or -race)
[...]

[...]
0.071 + cd /terraform                                                                                                                                                                                                                       
0.072 + tofu providers lock                                                                                                                                                                                                                 
1.465 - Fetching terraform-provider-openstack/openstack 3.4.0 for linux_arm64...                                                                                                                                                            
3.839 - Retrieved terraform-provider-openstack/openstack 3.4.0 for linux_arm64 (signed, key ID 4F80527A391BEFD2)                                                                                                                            
3.889 - Fetching gitlabhq/gitlab 18.6.1 for linux_arm64...
4.460 runtime: marked free object in span 0x4000475adc20, elemsize=6144 freeindex=2 (bad use of unsafe.Pointer or having race conditions? try -d=checkptr or -race)
[...]
Reproduce
  1. Clone the repo on https://github.com/sitsofe/tofu-crash or create the following files:

Dockerfile

# syntax=docker/dockerfile:1
FROM alpine:latest

RUN set -eux && \
    wget https://get.opentofu.org/install-opentofu.sh -O install-opentofu.sh && \
    sh install-opentofu.sh --install-method standalone --skip-verify --opentofu-version 1.11.1 && \
    rm -f install-opentofu.sh
COPY terraform /terraform/
ARG CACHE_BUSTER_VERSION=unknown
RUN set -eux; \
    \
    cd /terraform; \
    tofu providers lock

test-tofu.sh

#!/bin/bash
docker buildx create --name tofu-crash-builder --driver docker-container --bootstrap

if [[ $(arch) == "x86_64" ]]; then
  platform="linux/arm64"
else
  platform="linux/amd64"
fi

r=0
i=0
while [[ $r -eq 0 ]]; do
    i=$(( i + 1 ))
    echo $i
    docker buildx build --file Dockerfile --builder tofu-crash-builder --platform ${platform} -t tofu-crash --build-arg CACHE_BUSTER_VERSION="$(date +%s)" .
    r=$?
done
echo "$(date) Performed $i loop(s)"

terraform/main.tf

variable "mirror" {
  type    = string
  default = "mirror"
}

resource "openstack_compute_instance_v2" "instance_1" {
  name = "name"
  user_data = templatefile(
    "user_data.tftpl",
    {
      mirror = "${var.mirror}"
    }
  )
}

terraform/terraform.tf

terraform {
  required_providers {
    openstack = {
      source  = "terraform-provider-openstack/openstack"
      version = "~> 3.4.0"
    }
    gitlab = {
      source  = "gitlabhq/gitlab"
      version = "~> 18.6.1"
    }
  }
}

terraform/user_data.tftpl

(It's an empty file)

  1. Run
bash test-tofu.sh
Expected behavior

To be able to do 500 loops without error (if the issue is not hit you will have to manually kill the script as it loops until the exit code is not 0).

docker version
Client:
 Version:           29.1.3
 API version:       1.52
 Go version:        go1.25.5
 Git commit:        f52814d
 Built:             Fri Dec 12 14:48:46 2025
 OS/Arch:           darwin/arm64
 Context:           desktop-linux

Server: Docker Desktop 4.55.0 (213807)
 Engine:
  Version:          29.1.3
  API version:      1.52 (minimum version 1.44)
  Go version:       go1.25.5
  Git commit:       fbf3ed2
  Built:            Fri Dec 12 14:50:40 2025
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          v2.2.0
  GitCommit:        1c4457e00facac03ce1d75f7b6777a7a851e5c41
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb8
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0


Client: Docker Engine - Community                                                                                                                                                                                                           
 Version:           29.1.3
 API version:       1.52
 Go version:        go1.25.5
 Git commit:        f52814d
 Built:             Fri Dec 12 14:49:32 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.1.3
  API version:      1.52 (minimum version 1.44)
  Go version:       go1.25.5
  Git commit:       fbf3ed2
  Built:            Fri Dec 12 14:49:32 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v2.2.1
  GitCommit:        dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb8
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0


Client: Docker Engine - Community
 Version:           29.1.3
 API version:       1.52
 Go version:        go1.25.5
 Git commit:        f52814d
 Built:             Fri Dec 12 14:50:13 2025
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          29.1.3
  API version:      1.52 (minimum version 1.44)
  Go version:       go1.25.5
  Git commit:       fbf3ed2
  Built:            Fri Dec 12 14:50:13 2025
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          v2.2.1
  GitCommit:        dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc:
  Version:          1.3.4
  GitCommit:        v1.3.4-0-gd6d73eb8
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
docker info
Client:
 Version:    29.1.3
 Context:    desktop-linux
 Debug Mode: false
 Plugins:
  ai: Docker AI Agent - Ask Gordon (Docker Inc.)
    Version:  v1.17.1
    Path:     /Users/s/.docker/cli-plugins/docker-ai
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.30.1-desktop.1
    Path:     /Users/s/.docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.40.3-desktop.1
    Path:     /Users/s/.docker/cli-plugins/docker-compose
  debug: Get a shell into any image or container (Docker Inc.)
    Version:  0.0.45
    Path:     /Users/s/.docker/cli-plugins/docker-debug
  desktop: Docker Desktop commands (Docker Inc.)
    Version:  v0.2.0
    Path:     /Users/s/.docker/cli-plugins/docker-desktop
  extension: Manages Docker extensions (Docker Inc.)
    Version:  v0.2.31
    Path:     /Users/s/.docker/cli-plugins/docker-extension
  init: Creates Docker-related starter files for your project (Docker Inc.)
    Version:  v1.4.0
    Path:     /Users/s/.docker/cli-plugins/docker-init
  mcp: Docker MCP Plugin (Docker Inc.)
    Version:  v0.34.0
    Path:     /Users/s/.docker/cli-plugins/docker-mcp
  model: Docker Model Runner (Docker Inc.)
    Version:  v1.0.2
    Path:     /Users/s/.docker/cli-plugins/docker-model
  offload: Docker Offload (Docker Inc.)
    Version:  v0.5.33
    Path:     /Users/s/.docker/cli-plugins/docker-offload
  pass: Docker Pass Secrets Manager Plugin (beta) (Docker Inc.)
    Version:  v0.0.21
    Path:     /Users/s/.docker/cli-plugins/docker-pass
  sandbox: Docker Sandbox (Docker Inc.)
    Version:  v0.6.0
    Path:     /Users/s/.docker/cli-plugins/docker-sandbox
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc.)
    Version:  0.6.0
    Path:     /Users/s/.docker/cli-plugins/docker-sbom
  scout: Docker Scout (Docker Inc.)
    Version:  v1.18.3
    Path:     /Users/s/.docker/cli-plugins/docker-scout

Server:
 Containers: 4
  Running: 3
  Paused: 0
  Stopped: 1
 Images: 18
 Server Version: 29.1.3
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Discovered Devices:
  cdi: docker.com/gpu=webgpu
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 1c4457e00facac03ce1d75f7b6777a7a851e5c41
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.12.54-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: aarch64
 CPUs: 12
 Total Memory: 11.18GiB
 Name: docker-desktop
 ID: f05444e5-6925-4851-a78a-0c716536d06d
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 HTTP Proxy: http.docker.internal:3128
 HTTPS Proxy: http.docker.internal:3128
 No Proxy: hubproxy.docker.internal
 Labels:
  com.docker.desktop.address=unix:///Users/s/Library/Containers/com.docker.docker/Data/docker-cli.sock
 Experimental: false
 Insecure Registries:
  hubproxy.docker.internal:5555
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables


Client: Docker Engine - Community
 Version:    29.1.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.30.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.0.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  model: Docker Model Runner (Docker Inc.)
    Version:  v1.0.6
    Path:     /usr/libexec/docker/cli-plugins/docker-model

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 189
 Server Version: 29.1.3
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.11.0-17-generic
 Operating System: Ubuntu 24.04.2 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.62GiB
 Name: stest2
 ID: f0f46180-0f69-4efa-98a1-a1d5baf982fd
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables


Client: Docker Engine - Community
 Version:    29.1.3
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.30.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v5.0.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 29.1.3
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 CDI spec directories:
  /etc/cdi
  /var/run/cdi
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: dea7da592f5d1d2b7755e3a161be07f43fad8f75
 runc version: v1.3.4-0-gd6d73eb8
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.11.0-17-generic
 Operating System: Ubuntu 24.04.3 LTS
 OSType: linux
 Architecture: aarch64
 CPUs: 2
 Total Memory: 5.769GiB
 Name: stest3
 ID: e4b9908f-4b99-4f84-99a9-cdd43fc83e4e
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  ::1/128
  127.0.0.0/8
 Live Restore Enabled: false
 Firewall Backend: iptables
Additional Info
  • The problem only occurs under emulation - native (same platform as host) Docker builds will never have a problem
  • The problem happens on Docker Desktop on macOS 15 and Docker Engine on Ubuntu 24.04 (x86_64 and aarch64 hosts)
  • tofu typically crashes within 30 loops when the host platform is arm64 and amd64 is the emulated platform but crashes also happen the other way around (arm64 being emulated on amd64) but the loops required can be far more (e.g. over 200)
  • On Linux hosts with Docker Engine installed, if QEMU has been installed (e.g. via the distro package (apt-get install qemu-user-static) or via docker run --privileged --rm tonistiigi/binfmt --install all) then the problem still occurs (even though I can see the non-buildkit QEMU being used in ps ax). However when a builder (--builder tofu-crash-builder) is NOT used, even though emulation will still take place the problem will NOT occur (this is why I'm filing the issue here first)!
  • Trying to slim the terraform files down further can stop the issue from occurring
  • This problem is not https://github.com/golang/go/issues/69255 because it doesn't happen every time and tofu 1.11.2 has been compiled with Go 1.25.3

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running bash test-tofu.sh with the supplied Dockerfile, terraform files, and custom docker buildx builder to reproduce the arm64/amd64 emulation crash. Compare runs with and without the custom builder and inspect the reported Go runtime failure. Done means identifying and fixing the cause so the reproduction completes 500 loops without an error.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
build-system, devops, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.