build with `--no-cache` always invalidates `COPY` and `ADD`
Open
Nobody has claimed this yet.
kind/enhancement
status/needs-investigation
- Dominant language
- Go
- Stars
- 10.3k
- Forks
- 1.5k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 48
Description
Opening this issue because I was writing some examples to illustrate how BuildKit's --no-cache (unlike the classic builder) does not disable cache, but instead verifies cache when building images.
My expectation here was that steps like FROM, WORKDIR, LABEL, ENV will still use the cache if no changes were detected, and COPY and ADD can be cached if the source didn't change. For ADD from a remote source, BuildKit verifies if the remote content didn't change. RUN is always expected to be executed again, because executing commands can never be assumed to be idempotent (such as the RUN date in my example).
⚠️ It's possible that this was by design: a COPY will affect the parent directory's atime / mtime, so repeating a COPY even with the same content and parent state is not idempotent. However, the question is if we consider it to be an idempotent action (and perhaps this is a feature request for a more granular --nocache ("force", or "keep-timestamps" or ....?))
From a conversation on Slack with @tonistiigi and @crazy-max, where I wondered if this was intentional (i.e., forcing the (parent dir) timestamps to change);
I would expect only the
RUN dateto need to run again
...
I guess, theoretically this could be more safe as timestamps are not in theCOPYchecksum, so you could get a different result. but I don’t remember intentional change making copy and workdir to have a different behavior
I reproduced on both docker 24.0.7 (BuildKit 0.11.x) and 25.0.0-beta.1 (Buildkit 0.12.x);
24.0.7
docker version
Client: Docker Engine - Community
Version: 24.0.7
API version: 1.43
Go version: go1.20.10
Git commit: afdd53b
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 24.0.7
API version: 1.43 (minimum version 1.12)
Go version: go1.20.10
Git commit: 311b9ff
Built: Thu Oct 26 09:07:41 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.25
GitCommit: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
runc:
Version: 1.1.10
GitCommit: v1.1.10-0-g18a0cb0
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Client: Docker Engine - Community
Version: 24.0.7
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.21.0
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 9
Server Version: 24.0.7
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
runc version: v1.1.10-0-g18a0cb0
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.15.0-67-generic
Operating System: Ubuntu 22.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 969.4MiB
Name: ubuntu-22
ID: 166db08e-c4c5-47eb-b4a8-5dae586e2778
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
25.0.0-beta.1
Client: Docker Engine - Community
Version: 25.0.0-beta.1
API version: 1.44
Go version: go1.21.3
Git commit: 2b521e4
Built: Mon Nov 13 16:51:10 2023
OS/Arch: linux/amd64
Context: default
Server: Docker Engine - Community
Engine:
Version: 25.0.0-beta.1
API version: 1.44 (minimum version 1.12)
Go version: go1.21.3
Git commit: 6af7d6e
Built: Mon Nov 13 16:49:37 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.25
GitCommit: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
runc:
Version: 1.1.10
GitCommit: v1.1.10-0-g18a0cb0
docker-init:
Version: 0.19.0
GitCommit: de40ad0
Client: Docker Engine - Community
Version: 25.0.0-beta.1
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.23.0
Path: /usr/libexec/docker/cli-plugins/docker-compose
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 15
Server Version: 25.0.0-beta.1
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
runc version: v1.1.10-0-g18a0cb0
init version: de40ad0
Security Options:
seccomp
Profile: builtin
cgroupns
Kernel Version: 5.14.0-325.el9.x86_64
Operating System: CentOS Stream 9
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 771.4MiB
Name: centos-9
ID: 4636775d-ec3f-41fc-9c0a-cd9ffde76ed2
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Prepare
- create a fresh directory
- create a
not-changed.txt - create a Dockerfile
# syntax=docker/dockerfile:1
FROM ubuntu:latest
WORKDIR /src
LABEL foo=bar
ENV hello=world
COPY not-changed.txt .
ADD https://download.docker.com/linux/ubuntu/gpg .
RUN ls -la /src/
RUN date
mkdir repro && cd repro
echo hello > not-changed.txt
cat > Dockerfile <<'EOF'
# syntax=docker/dockerfile:1
FROM ubuntu:latest
WORKDIR /src
LABEL foo=bar
ENV hello=world
COPY not-changed.txt .
ADD https://download.docker.com/linux/ubuntu/gpg .
RUN ls -la /src/
RUN date
EOF
Initial build, with --no-cache:
docker build --no-cache .
[+] Building 20.1s (14/14) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.4s
=> => transferring dockerfile: 231B 0.0s
=> [internal] load .dockerignore 0.5s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1 1.6s
=> docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021 1.8s
=> => resolve docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021 0.3s
=> => sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021 8.40kB / 8.40kB 0.0s
=> => sha256:657fcc512c7369f4cb3d94ea329150f8daf626bc838b1a1e81f1834c73ecc77e 482B / 482B 0.0s
=> => sha256:a17ee7fff8f5e97b974f5b48f51647d2cf28d543f2aa6c11aaa0ea431b44bb89 1.27kB / 1.27kB 0.0s
=> => sha256:9d9c93f4b00be908ab694a4df732570bced3b8a96b7515d70ff93402179ad232 11.80MB / 11.80MB 0.8s
=> => extracting sha256:9d9c93f4b00be908ab694a4df732570bced3b8a96b7515d70ff93402179ad232 0.3s
=> [internal] load metadata for docker.io/library/ubuntu:latest 1.5s
=> [1/6] FROM docker.io/library/ubuntu:latest@sha256:2b7412e6465c3c7fc5bb21d3e6f1917c167358449fecac8176c6e496e5c1f05f 5.7s
=> => resolve docker.io/library/ubuntu:latest@sha256:2b7412e6465c3c7fc5bb21d3e6f1917c167358449fecac8176c6e496e5c1f05f 0.8s
=> => sha256:2b7412e6465c3c7fc5bb21d3e6f1917c167358449fecac8176c6e496e5c1f05f 1.13kB / 1.13kB 0.0s
=> => sha256:c9cf959fd83770dfdefd8fb42cfef0761432af36a764c077aed54bbc5bb25368 424B / 424B 0.0s
=> => sha256:e4c58958181a5925816faa528ce959e487632f4cfd192f8132f71b32df2744b4 2.30kB / 2.30kB 0.0s
=> => sha256:aece8493d3972efa43bfd4ee3cdba659c0f787f8f59c82fb3e48c87cbb22a12e 29.54MB / 29.54MB 0.9s
=> => extracting sha256:aece8493d3972efa43bfd4ee3cdba659c0f787f8f59c82fb3e48c87cbb22a12e 2.2s
=> https://download.docker.com/linux/ubuntu/gpg 1.7s
=> [internal] load build context 1.3s
=> => transferring context: 48B 0.0s
=> [2/6] WORKDIR /src 0.7s
=> [3/6] COPY not-changed.txt . 0.8s
=> [4/6] ADD https://download.docker.com/linux/ubuntu/gpg . 1.0s
=> [5/6] RUN ls -la /src/ 1.1s
=> [6/6] RUN date 1.3s
=> exporting to image 1.4s
=> => exporting layers 1.3s
=> => writing image sha256:188e2cc3fe32f7cc98211823ad7eb15df16698bb7eb253a699ef6e25ad397b1b 0.2s
Repeated build: COPY and ADD don't show CACHED (despite no changes for both);
docker build --no-cache .
[+] Building 11.2s (14/14) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.3s
=> => transferring dockerfile: 231B 0.0s
=> [internal] load .dockerignore 0.5s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1 1.1s
=> CACHED docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:latest 0.5s
=> [1/6] FROM docker.io/library/ubuntu:latest@sha256:2b7412e6465c3c7fc5bb21d3e6f1917c167358449fecac8176c6e496e5c1f05f 0.0s
=> CACHED https://download.docker.com/linux/ubuntu/gpg 0.0s
=> CACHED [2/6] WORKDIR /src 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 36B 0.0s
=> [3/6] COPY not-changed.txt . 0.4s
=> [4/6] ADD https://download.docker.com/linux/ubuntu/gpg . 0.8s
=> [5/6] RUN ls -la /src/ 0.9s
=> [6/6] RUN date 0.9s
=> exporting to image 2.9s
=> => exporting layers 2.8s
=> => writing image sha256:3dbf0b218ed6503b04cd744f1b7f1b5afed0464145c84b40614f4121c82ee519 0.1s
docker build --no-cache .
[+] Building 10.6s (14/14) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.1s
=> => transferring dockerfile: 231B 0.0s
=> [internal] load .dockerignore 0.2s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1 1.8s
=> CACHED docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:latest 1.2s
=> [1/6] FROM docker.io/library/ubuntu:latest@sha256:2b7412e6465c3c7fc5bb21d3e6f1917c167358449fecac8176c6e496e5c1f05f 0.0s
=> CACHED https://download.docker.com/linux/ubuntu/gpg 0.1s
=> CACHED [2/6] WORKDIR /src 0.0s
=> [internal] load build context 0.3s
=> => transferring context: 36B 0.0s
=> [3/6] COPY not-changed.txt . 1.3s
=> [4/6] ADD https://download.docker.com/linux/ubuntu/gpg . 0.7s
=> [5/6] RUN ls -la /src/ 1.0s
=> [6/6] RUN date 1.1s
=> exporting to image 1.0s
=> => exporting layers 0.9s
=> => writing image sha256:8b48095edef742867fccb24ab42e8a12f5b5070ecaa2a402e86e69bec353a10f 0.0s
To verify if the issue was due to ALL builds so far were executed with --no-cache, I also tried building with cache enabled, and --no-cache after that.
Without --no-cache, the output shows CACHED, and the digest does not change:
docker build .
[+] Building 1.3s (14/14) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 231B 0.0s
=> [internal] load .dockerignore 0.1s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1 0.4s
=> CACHED docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:latest 0.4s
=> [1/6] FROM docker.io/library/ubuntu:latest@sha256:2b7412e6465c3c7fc5bb21d3e6f1917c167358449fecac8176c6e496e5c1f05f 0.0s
=> https://download.docker.com/linux/ubuntu/gpg 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 36B 0.0s
=> CACHED [2/6] WORKDIR /src 0.0s
=> CACHED [3/6] COPY not-changed.txt . 0.0s
=> CACHED [4/6] ADD https://download.docker.com/linux/ubuntu/gpg . 0.0s
=> CACHED [5/6] RUN ls -la /src/ 0.0s
=> CACHED [6/6] RUN date 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:dc68bc71a269a4c99eb217e50e5e7b84a0c2c05d0ca3bd2f737c7a1287838315 0.0s
docker build .
[+] Building 1.2s (14/14) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.0s
=> => transferring dockerfile: 231B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1 0.4s
=> CACHED docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:latest 0.4s
=> [1/6] FROM docker.io/library/ubuntu:latest@sha256:2b7412e6465c3c7fc5bb21d3e6f1917c167358449fecac8176c6e496e5c1f05f 0.0s
=> https://download.docker.com/linux/ubuntu/gpg 0.0s
=> [internal] load build context 0.0s
=> => transferring context: 36B 0.0s
=> CACHED [2/6] WORKDIR /src 0.0s
=> CACHED [3/6] COPY not-changed.txt . 0.0s
=> CACHED [4/6] ADD https://download.docker.com/linux/ubuntu/gpg . 0.0s
=> CACHED [5/6] RUN ls -la /src/ 0.0s
=> CACHED [6/6] RUN date 0.0s
=> exporting to image 0.0s
=> => exporting layers 0.0s
=> => writing image sha256:dc68bc71a269a4c99eb217e50e5e7b84a0c2c05d0ca3bd2f737c7a1287838315 0.0s
Building with --no-cache after this doesn't cache COPY and ADD;
docker build --no-cache .
[+] Building 4.1s (14/14) FINISHED docker:default
=> [internal] load build definition from Dockerfile 0.6s
=> => transferring dockerfile: 231B 0.0s
=> [internal] load .dockerignore 0.6s
=> => transferring context: 2B 0.0s
=> resolve image config for docker.io/docker/dockerfile:1 0.5s
=> CACHED docker-image://docker.io/docker/dockerfile:1@sha256:ac85f380a63b13dfcefa89046420e1781752bab202122f8f50032edf31be0021 0.0s
=> [internal] load metadata for docker.io/library/ubuntu:latest 0.6s
=> [1/6] FROM docker.io/library/ubuntu:latest@sha256:2b7412e6465c3c7fc5bb21d3e6f1917c167358449fecac8176c6e496e5c1f05f 0.0s
=> CACHED https://download.docker.com/linux/ubuntu/gpg 0.0s
=> CACHED [2/6] WORKDIR /src 0.0s
=> [internal] load build context 0.1s
=> => transferring context: 36B 0.0s
=> [3/6] COPY not-changed.txt . 0.1s
=> [4/6] ADD https://download.docker.com/linux/ubuntu/gpg . 0.1s
=> [5/6] RUN ls -la /src/ 0.4s
=> [6/6] RUN date 0.5s
=> exporting to image 0.4s
=> => exporting layers 0.4s
=> => writing image sha256:fd6b982e45526d93057f642b4de54505208c50dbe2215fae44e1e9a5d8592f60 0.0s
I also checked if it was perhaps only a presentation issue (maybe the COPY and ADD were not actually executed). For that I ran build with --progress=plain to check the output of ls -la /src/, which shows that the /src/ directory modified date was updated, so the file was copied again;
docker build --no-cache --progress=plain .
...
#12 [5/6] RUN ls -la /src/
#12 0.396 total 16
#12 0.396 drwxr-xr-x 1 root root 4096 Nov 23 09:11 .
#12 0.396 drwxr-xr-x 1 root root 4096 Nov 23 09:11 ..
#12 0.396 -rw------- 1 root root 3817 Nov 21 08:57 gpg
#12 0.396 -rw-r--r-- 1 root root 6 Nov 23 08:48 not-changed.txt
#12 DONE 0.5s
#13 [6/6] RUN date
#13 0.280 Thu Nov 23 09:11:40 UTC 2023
#13 DONE 0.3s
docker build --no-cache --progress=plain .
...
#12 [5/6] RUN ls -la /src/
#12 0.879 total 16
#12 0.879 drwxr-xr-x 1 root root 4096 Nov 23 09:12 .
#12 0.879 drwxr-xr-x 1 root root 4096 Nov 23 09:12 ..
#12 0.879 -rw------- 1 root root 3817 Nov 21 08:57 gpg
#12 0.879 -rw-r--r-- 1 root root 6 Nov 23 08:48 not-changed.txt
#12 DONE 1.3s
#13 [6/6] RUN date
#13 0.744 Thu Nov 23 09:12:36 UTC 2023
#13 DONE 0.9s
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the issue with the shown Dockerfile and docker build --no-cache ., focusing on the WORKDIR, COPY, ADD, and RUN steps. Compare which steps execute across repeated builds and determine the intended cache semantics for COPY and ADD; done should include a settled behavior and coverage for the reproduced case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, dockerfile, go
- Domain
- build-system, tooling
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100