concourse / concourse/docker-image-resource

dockerd crashes ~18s after startup in v1.13.1 due to BuildKit healthcheck fatal failure (Docker 29.x)

Open
#385 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Go
Stars
162
Forks
250
PR merge metrics
No merged PRs in 30d

Description

Describe the bug

Summary

Starting with the Docker 29.x image bundled in v1.13.1, dockerd reliably crashes
~18 seconds after startup, causing all put steps to fail. The root cause is
BuildKit's internal gRPC session healthcheck failing fatally due to a
"only one connection allowed" socket transport restriction in the containerized
Concourse worker environment.

Symptoms

  • put steps using docker-image-resource fail consistently across retries
  • Hijacking the container shows /var/run/docker.sock does not exist (daemon has exited)
  • start_docker appears to succeed (prints "waiting for docker to come up...") but
    the daemon exits shortly after

Root cause

/tmp/docker.log from a failed task shows:

INFO  API listen on /var/run/docker.sock              ← daemon up
WARN  healthcheck failed: "only one connection allowed"
ERROR healthcheck failed fatally: session healthcheck failed fatally
INFO  Processing signal 'terminated'                  ← daemon exits ~18s after start

Docker 29.x has BuildKit deeply integrated into the daemon (always-on). BuildKit's
internal gRPC session healthcheck attempts to open a second connection on the Unix
socket, but the containerized environment only permits one connection at a time.
The healthcheck fails fatally and brings down dockerd with it.

This did not occur in older Docker versions bundled with v1.12.4 where BuildKit was
not integrated at this level.

Environment

  • docker-image-resource version: v1.13.1
  • Docker version in image: 29.5.1
  • Base OS: Wolfi (containerized)
  • Kernel: 5.15.0-181-generic
  • Cgroup: v1

Workaround

Pin to v1.12.4 in your pipeline:

resource_types:
- name: docker-image
  type: registry-image
  source:
    repository: concourse/docker-image-resource
    tag: 1.12.4

Possible fixes

In assets/common.sh, writing a daemon.json disabling the BuildKit integration
before starting dockerd, or passing the flag directly:

Option A — daemon.json

echo '{"features":{"buildkit":false}}' > /etc/docker/daemon.json

Option B — flag

dockerd --feature=buildkit=false ...

Note: setting DOCKER_BUILDKIT=0 is not sufficient in Docker 29.x as BuildKit is now integrated into the daemon itself, not just the CLI. So I think the docker_buildkit: false will no longer work.

Reproduction steps
  1. Use v1.13.1 to build and push an iamge
Expected behavior

Push succeeds as in older registry image resources

Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with assets/common.sh and the start_docker path to understand how dockerd is launched and configured. Reproduce the v1.13.1 failure with a docker-image-resource put, then verify the selected daemon configuration or flag keeps dockerd running and allows the put to succeed without regressing the older setup.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, shell
Domain
devops, infrastructure
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.