"docker builder prune" can cause "docker build" to hang
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 6.1k
- Forks
- 2.2k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 43
Description
Description
We're using Jenkins with dockerfiles, and occasionally we run docker builder prune when the build cache gets too large.
Recently we've noticed that when we do that, any running docker build commands will hang and after 10 minutes Jenkins will say:
12:08:45 wrapper script does not seem to be touching the log file in /var/jenkins_home/workspace/t-fix-client_release@tmp/durable-a3bfd329
12:08:45 (JENKINS-48300: if on an extremely laggy filesystem, consider -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400)
and the job fails.
Reproduce
docker build &docker builder prune
Expected behavior
The documentation said docker builder prune will delete dangling build cache entries, which would suggest entries that are not currently being used, and so no current builds should be affected.
docker version
Client:
Version: 20.10.12
API version: 1.41
Go version: go1.16.2
Git commit: 20.10.12-0ubuntu2~18.04.1
Built: Fri Oct 21 08:25:48 2022
OS/Arch: linux/amd64
Context: default
Experimental: true
Server:
Engine:
Version: 20.10.12
API version: 1.41 (minimum version 1.12)
Go version: go1.16.2
Git commit: 20.10.12-0ubuntu2~18.04.1
Built: Mon Apr 4 20:53:56 2022
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.5.9-0ubuntu1~18.04.2
GitCommit:
runc:
Version: 1.1.0-0ubuntu1~18.04.1
GitCommit:
docker-init:
Version: 0.19.0
GitCommit:
docker info
Client:
Context: default
Debug Mode: false
Server:
Containers: 21
Running: 10
Paused: 0
Stopped: 11
Images: 9
Server Version: 20.10.12
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version:
runc version:
init version:
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-200-generic
Operating System: Ubuntu 18.04.6 LTS
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 125.9GiB
Name: jenkins2.dev.netskrt.org
ID: A45J:HV27:WVNG:UPWL:IAYN:26IO:DUJY:UJO7:IL3H:N2YN:QYJJ:QWM6
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Additional Info
This is on Ubuntu 18.04.6 LTS, docker installed with apt install docker.io, and it's the latest version in the Ubuntu repository.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the hang with a background docker build followed by docker builder prune, using the reported Docker 20.10.12 environment as a reference. Trace the CLI and builder path handling these commands and verify that pruning no longer blocks or hangs an active build, with a regression test covering the sequence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100