actions / actions/runner

Docker socket "broken pipe" failures: runner v2.329.0 + Docker v29.0.1 in Kubernetes

Open
#4,117 4 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
C#
Stars
6.3k
Forks
1.4k
Avg merge
1d 16h
Merged PRs (30d)
24

Description

Describe the bug
We are experiencing intermittent Docker socket communication failures across multiple self-hosted GitHub Actions runner scale sets running in Kubernetes with Docker-in-Docker (dind).

Issue started immediately after upgrading from runner v2.328.0 → v2.329.0

The Problem:

  1. Upgraded runner from v2.328.0 to v2.329.0 around late October 2025
  2. Errors began appearing on October 31, 2025 (shortly after upgrade)
  3. No issues with v2.328.0 - runners were stable before the upgrade
  4. Random workflow failures with "broken pipe" errors when the runner tries to attach to container exec stdout streams
  5. 5-10% of jobs fail randomly during peak hours

Error Pattern:
The dind container logs show socket write failures when trying to communicate with /var/run/docker.sock. All scale sets experience errors simultaneously during spike events, suggesting this is more than just resource exhaustion.

Runner Version and Platform
Runner:
Version: v2.329.0
Deployment Method: actions-runner-controller (gha-runner-scale-set)
Controller Version: 0.12.1
Chart Version: 0.12.1

Kubernetes:
Platform: Amazon EKS
EKS Version: 1.33
EKS AMI: 1.33.5-20251023 (October 2025 release)

Docker (in dind container):
Docker Engine: 29.0.1 (built November 14, 2025)
API version: 1.52
containerd: v2.1.5
runc: 1.3.3
Image: docker:dind (latest tag from Docker Hub)

Runner Logs
Error Messages - Primary Error (from dind container stderr): Error running exec <container_id> in container: exec attach failed:
error attaching stdout stream: write unix /var/run/docker.sock->@: write: broken pipe

Secondary Error: http2: server: error reading preface from client @: read unix /var/run/docker.sock->@: read: connection reset by peer

Sample Logs from dind Container
2025-11-14T07:01:11.063364452Z level=error msg="Error running exec 2bdbf255470b5a38 in container: exec attach failed: error attaching stdout stream: write unix /var/run/docker.sock->@: write: broken pipe"
2025-11-14T07:04:31.631887068Z level=error msg="Error running exec 06cffe9160ae9 in container: exec attach failed: error attaching stdout stream: write unix /var/run/docker.sock->@: write: broken pipe"
2025-11-14T07:04:34.408Z level=warning msg="ShouldRestart failed, container will not be restarted" daemonShuttingDown=false error="" execDuration=3m10.311722009s exitStatus="{1 2025-11-14 07:04:34.390005475 +0000 UTC}" hasBeenManuallyStopped=true restartCount=0
2025-11-14T07:04:54.908Z http2: server: error reading preface from client @: read unix /var/run/docker.sock->@: read: connection reset by peer

Runner and Worker's Diagnostic Logs
No errors appear in the runner container logs. All errors are only visible in the dind (Docker-in-Docker) container stderr logs. The runner container shows normal operation.

Questions

  1. What changed in v2.329.0 related to Docker socket handling?
  2. Is there a known regression in v2.329.0 with dind setups?
  3. Are there recommended socket configurations for v2.329.0 with dind?
  4. Could Docker v29.0.1 (Nov 14, 2025) + runner v2.329.0 combination be problematic?

We would appreciate guidance on!

Thank you

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by comparing runner v2.328.0 and v2.329.0 behavior in the described Docker-in-Docker Kubernetes setup, then inspect the dind container stderr logs for the broken-pipe and connection-reset sequence. Check the runner and Docker socket interaction alongside Docker v29.0.1. Done means identifying whether this is a runner regression, a Docker compatibility issue, or a socket configuration problem, with reproducible evidence or validated guidance.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, docker, github-actions, kubernetes
Domain
ci-cd, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.