"Removal of container already in progress" and "cannot start a container that has stopped"

Description

We are using Docker in GitHub Actions Runners using the summerwind/actions-runner-dind:ubuntu-22.04 runner image in Kubernetes. This is the so-called Docker-in-Docker variant.

Containers started either by a workflow or by a human agent with console access to the runner pod will randomly exit with an error from the Docker daemon as though multiple conflicting lifecycle events had been received. The error messages always imply that simultaneous delete and start events occurred at the end of the container's lifecycle, even if the --rm argument was not passed to the Docker CLI. We suspect this to be a race condition in Docker's lifecycle management.

This can occur within shared actions (e.g., https://github.com/docker/setup-qemu-action by Docker) that do nothing but pull and execute a container with docker run.

The problem is plaguing the CI workflows I manage with unreliability, occurring multiple times every day.

Reproduce

1. /usr/local/bin/docker run --rm --privileged tonistiigi/binfmt:latest --install all (as done by docker/setup-qemu-action; this is a trivial and functional example, but it will often happen at the end of much longer Dockerized CI/CD workflows)

If the error occurs, it looks like this to the person running it:

  write exec fifo /proc/self/fd/5: broken pipe
  docker: Error response from daemon: OCI runtime start failed: cannot start a container that has stopped: unknown.

A subsequent attempt can succeed.

Expected behavior

Container should exit successfully and return its expected output. In the example of the docker/setup-qemu-action workflow, stdout would receive e.g.:

  installing: riscv64 OK
  installing: arm64 OK
  installing: arm OK
  installing: s390x OK
  installing: ppc64le OK
  installing: mips64le OK
  installing: mips64 OK
  {
    "supported": [
      "linux/amd64",
      "linux/arm64",
      "linux/riscv64",
      "linux/ppc64le",
      "linux/s390x",
      "linux/386",
      "linux/mips64le",
      "linux/mips64",
      "linux/arm/v7",
      "linux/arm/v6"
    ],
    "emulators": [
      "qemu-aarch64",
      "qemu-arm",
      "qemu-mips64",
      "qemu-mips64el",
      "qemu-ppc64le",
      "qemu-riscv64",
      "qemu-s390x"
    ]
  }

docker version

Client:
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:04:00 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:05:28 2023
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.7.6
  GitCommit:        091922f03c2762540fd057fba91260237ff86acb
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

docker info

Client:
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc.)
    Version:  v2.23.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 091922f03c2762540fd057fba91260237ff86acb
 runc version: v1.1.9-0-gccaecfc
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.10.199-190.747.amzn2.x86_64
 Operating System: Ubuntu 22.04.3 LTS (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.27GiB
 Name: actions-runner-fwx8v-5wl8w
 ID: dd99afce-b8e8-40d9-bbfa-89edba527945
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Additional Info

Docker daemon debug logs.

The representative signature consists of the following two lines:

error removing container container=9eb9157e23b8745ade7dbb666e0aab051030d93879b553df852ee9d956a6ae36 error="removal of container 9eb9157e23b8745ade7dbb666e0aab051030d93879b553df852ee9d956a6ae36 is already in progress"
Handler for POST /v1.43/containers/9eb9157e23b8745ade7dbb666e0aab051030d93879b553df852ee9d956a6ae36/start returned error: OCI runtime start failed: cannot start a container that has stopped: unknown

/etc/docker/daemon.json has experimental set to true because it is required for the option ip6tables:

{
  "dns": ["169.254.169.253", "fd00:ec2::253"],
  "experimental": true,
  "ip6tables": true,
  "debug": true,
  "log-driver": "journald"
}

Start by reproducing the failure with /usr/local/bin/docker run --rm --privileged tonistiigi/binfmt:latest --install all in the described Docker-in-Docker environment. Review the linked daemon debug log and the reported removal/start errors to trace the conflicting lifecycle events. Done means the command exits successfully and produces its expected output without either daemon error.