Multi-stage builds exhibit unexpected image size expansion

In some multi-stage builds, the occupied space of the finished image can expand unexpected and inappropriately.

Here is a minimum-working-example demo Dockerfile that causes the problem:

FROM alpine AS base
RUN apk add --no-cache coreutils

FROM base AS builder
WORKDIR /data
RUN echo "in builder" > /builder.txt
RUN dd if=/dev/urandom of=/data/bigfile.dat bs=1M count=64

FROM base
COPY --from=builder /data /data
RUN echo "test" > /file.txt

If I build this image without BuildKit enabled...

DOCKER_BUILDKIT=0 docker build -t bug .

the size of the finished image is 74.5MB:

fmillion:~/src/docker/bug$ docker images
REPOSITORY                           TAG                            IMAGE ID       CREATED             SIZE
bug                                  latest                         df30c93ed2b1   1 second ago        74.5MB

This is the correct expected size. The base Alpine image plus coreutils plus a 64MB random data file yields ~74MB, exactly the expected size.

However, if I build the image with Buildkit:

DOCKER_BUILDKIT=1 docker build -t bug .

the image comes out as 145MB:

fmillion:~/src/docker/bug$ docker images
REPOSITORY                           TAG                            IMAGE ID       CREATED             SIZE
bug                                  latest                         1aa4c9d2d9a5   3 minutes ago       149MB

See the docker history output:

fmillion:~/src/docker/bug$ docker history bug
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
3ec2b2d665b2   8 seconds ago    RUN /bin/sh -c echo "test" > /file.txt # bui…   74.5MB    buildkit.dockerfile.v0
<missing>      11 seconds ago   COPY /data /data/ # buildkit                    67.1MB    buildkit.dockerfile.v0
<missing>      15 seconds ago   RUN /bin/sh -c apk add --no-cache coreutils …   1.84MB    buildkit.dockerfile.v0
<missing>      3 months ago     /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      3 months ago     /bin/sh -c #(nop) ADD file:2a949686d9886ac7c…   5.54MB

Notice how the layer that adds a "test" file adds 74.5MB. Also note that 74.5MB is exactly the size of the original output image. Compare this with a run without BuildKit:

fmillion:~/src/docker/bug$ docker history bug
IMAGE          CREATED          CREATED BY                                      SIZE      COMMENT
f96a300976a6   3 seconds ago        /bin/sh -c echo "test" > /file.txt              5B
6480ff13497f   12 seconds ago       /bin/sh -c #(nop) COPY dir:7980da196687c5df0…   67.1MB
4f86b8c0ed8b   About a minute ago   /bin/sh -c apk add --no-cache coreutils         1.84MB
9c6f07244728   3 months ago         /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      3 months ago         /bin/sh -c #(nop) ADD file:2a949686d9886ac7c…   5.54MB

This is exactly the expected result. Note that the only difference is on the line where we echo "test" to a file in the root directory.

Here's what I theorize is happening. The COPY command, when copying from another container stage, is somehow creating a link to that stage, and then when the final container makes a change to the filesystem, the comparison is being made against the other stage, not the current stage. In other words, it looks like Docker sees that the build stage is somehow "different" from the final stage, and copies in the diff between the build stage and the final stage. Or, perhaps, it has to do with parallel builds.

The actual resulting image is correct - the file "file.txt" exists at the root, and the file "bigfile.dat" exists inside /data. However, the total image size is too large. Also a df -x / produces the expected total size of all files in the container.

In a more complex dockerfile with more steps, random steps in the final output file will show space used, including non-data consuming steps like WORKDIR. Here's the docker history of an app I'm developing with two stages of build:

fmillion:/home/fmillion/src/scanapp$ docker history scanapp:with_bk
IMAGE          CREATED        CREATED BY                                      SIZE      COMMENT
55cb5ed04c3d   2 hours ago    CMD ["python3" "/app/scan.py"]                  0B        buildkit.dockerfile.v0
<missing>      2 hours ago    RUN /bin/sh -c rm -rf /wheels # buildkit        99MB      buildkit.dockerfile.v0
<missing>      2 hours ago    RUN /bin/sh -c pip3 install --find-links /wh…   18.3MB    buildkit.dockerfile.v0
<missing>      2 hours ago    WORKDIR /app                                    1.27MB    buildkit.dockerfile.v0
<missing>      2 hours ago    COPY /wheels /wheels # buildkit                 3.18MB    buildkit.dockerfile.v0
<missing>      2 hours ago    COPY requirements.txt *.py /app/ # buildkit     38kB      buildkit.dockerfile.v0
<missing>      2 hours ago    RUN /bin/sh -c python3 -m ensurepip # buildk…   80.7MB    buildkit.dockerfile.v0
<missing>      2 hours ago    RUN /bin/sh -c apk add python3 gettext zlib …   58.1MB    buildkit.dockerfile.v0
<missing>      2 hours ago    RUN /bin/sh -c apk update && apk upgrade # b…   6.48MB    buildkit.dockerfile.v0
<missing>      3 months ago   /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>      3 months ago   /bin/sh -c #(nop) ADD file:2a949686d9886ac7c…   5.54MB

Of note in this example is that the WORKDIR step added ~1MB, and a step that executes an rm -rf added 99MB. Removing the rm -rf step makes the image come out the correct size - rm -rf is the last step prior to the image being finished. I do understand that executing rm won't actually remove files from the layers and they won't take up less space, but rm should not be adding space to the layers.

For another example, try adding RUN rm -rf /data to the minimum working Dockerfile at the very end. The image will still come out as 149MB, but with the correct result (no 64MB random file in /data).

Output of docker version and docker info:

fmillion:~$ docker version
Client:
 Version:           20.10.20
 API version:       1.41
 Go version:        go1.18.7
 Git commit:        9fdeb9c3de2f2d9f5799be373f27b2f9df44609d
 Built:             Wed Oct 19 03:04:31 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.20
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.7
  Git commit:       03df974ae9e6c219862907efdd76ec2e77ec930b
  Built:            Wed Oct 19 02:58:31 2022
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          v1.6.8
  GitCommit:        9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc:
  Version:          1.1.4
  GitCommit:        5fd4c4d144137e991c4acebb2146ab1483a97925
 docker-init:
  Version:          0.19.0
  GitCommit:

fmillion:~$ docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  compose: Docker Compose (Docker Inc., v2.6.0)

Server:
 Containers: 7
  Running: 4
  Paused: 0
  Stopped: 3
 Images: 22
 Server Version: 20.10.20
 Storage Driver: zfs
  Zpool: dockerpool
  Zpool Health: ONLINE
  Parent Dataset: dockerpool/docker
  Space Used By Parent: 22820552704
  Space Available: 11861979136
  Parent Quota: no
  Compression: zstd
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: 5fd4c4d144137e991c4acebb2146ab1483a97925
 init version:
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.15.77-0-lts
 Operating System: Alpine Linux v3.16
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 23.39GiB
 Name: skylake-testbox
 ID: BU3M:D5KG:NBIP:RHV7:3R7U:KDZQ:KCS2:EJFY:MY3X:ILR6:YSH2:5QFI
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: true
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Start by reproducing the minimum Dockerfile with BuildKit enabled and disabled, then compare docker history and the resulting image sizes. Investigate the stage-copy and subsequent filesystem-change behavior described in the report. Done means the final image remains correctly sized and later no-op or deletion steps do not report the earlier stage contents as newly added layer data.