docker / docker/cli

"docker builder prune" can cause "docker build" to hang

Open
#4,021 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

area/builder area/builder/buildkit kind/bug status/0-triage version/20.10
Dominant language
Go
Stars
6.1k
Forks
2.2k
Avg merge
1d 15h
Merged PRs (30d)
43

Description

Description

We're using Jenkins with dockerfiles, and occasionally we run docker builder prune when the build cache gets too large.

Recently we've noticed that when we do that, any running docker build commands will hang and after 10 minutes Jenkins will say:

12:08:45  wrapper script does not seem to be touching the log file in /var/jenkins_home/workspace/t-fix-client_release@tmp/durable-a3bfd329
12:08:45  (JENKINS-48300: if on an extremely laggy filesystem, consider -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400)

and the job fails.

Reproduce
  1. docker build &
  2. docker builder prune
Expected behavior

The documentation said docker builder prune will delete dangling build cache entries, which would suggest entries that are not currently being used, and so no current builds should be affected.

docker version
Client:
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.16.2
 Git commit:        20.10.12-0ubuntu2~18.04.1
 Built:             Fri Oct 21 08:25:48 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.2
  Git commit:       20.10.12-0ubuntu2~18.04.1
  Built:            Mon Apr  4 20:53:56 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.9-0ubuntu1~18.04.2
  GitCommit:        
 runc:
  Version:          1.1.0-0ubuntu1~18.04.1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:
docker info
Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 21
  Running: 10
  Paused: 0
  Stopped: 11
 Images: 9
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-200-generic
 Operating System: Ubuntu 18.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 32
 Total Memory: 125.9GiB
 Name: jenkins2.dev.netskrt.org
 ID: A45J:HV27:WVNG:UPWL:IAYN:26IO:DUJY:UJO7:IL3H:N2YN:QYJJ:QWM6
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support
Additional Info

This is on Ubuntu 18.04.6 LTS, docker installed with apt install docker.io, and it's the latest version in the Ubuntu repository.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the hang with a background docker build followed by docker builder prune, using the reported Docker 20.10.12 environment as a reference. Trace the CLI and builder path handling these commands and verify that pruning no longer blocks or hangs an active build, with a regression test covering the sequence.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
cli
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.