docker-container driver: `COPY --link --chown` unexpectedly changes parent dir ownership

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

Originally reported at BuildKit: https://github.com/moby/buildkit/issues/3912

Scenario: Adding content from a different image on DockerHub via COPY --link, and correcting the ownership with --chown=<uid>.

When building with the buildx driver docker-container, the parent directories (/var/lib) appear to also have had their ownership modified to the --chown value.

It's possible that this bug is related to how the feature(s) work when pulling an image from a registry, or when building an image locally with --load via the docker-container driver. The native docker driver does not build images with this ownership bug.

Expected behaviour

/var and /var/lib should not have their ownership changed since they already exist in the image, only /var/lib/clamav was copied over.

Actual behaviour

Parent directories ownership is changed from the COPY --link --chown.

Buildx version

github.com/docker/buildx v0.10.4 c513d34

Docker info

Client: Docker Engine - Community
 Version:    24.0.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.4
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 3
  Running: 3
  Paused: 0
  Stopped: 0
 Images: 25
 Server Version: 24.0.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.0-20-generic
 Operating System: Ubuntu 23.04
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 945.4MiB
 Name: vpc-ubu
 ID: 028ce824-5aaf-4d0b-97cc-c31018736f0f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Builders list

NAME/NODE DRIVER/ENDPOINT             STATUS  BUILDKIT PLATFORMS
con *     docker-container                             
  con0    unix:///var/run/docker.sock running v0.11.6  linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386
default   docker                                       
  default default                     running v0.11.6  linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386

Configuration

# syntax=docker.io/docker/dockerfile:1

FROM docker.io/debian:11-slim
RUN adduser --quiet --system --group --disabled-password --home /var/lib/clamav --no-create-home --uid 200 clamav
COPY --link --chown=200 --from=docker.io/clamav/clamav:latest /var/lib/clamav /var/lib/clamav

$ docker buildx create --driver docker-container --name con --use
$ docker buildx build -t test --load .
$ docker run --rm test ls -l /var | grep clamav

drwxr-xr-x 1 clamav clamav 4096 May 29 05:46 lib

Additional info

This bug seems potentially related to:

• https://github.com/moby/buildkit/issues/3602#issuecomment-1422869270
• https://github.com/moby/buildkit/issues/2414#issuecomment-943325055

We do have these current releases where you can observe this by pulling from the registry:

• mailserver/docker-mailserver:11.3.1: Dockerfile COPY --link line
• mailserver/docker-mailserver:12.0.0: Dockerfile COPY --link line with --chown=200 added

Pulling the v12 image or anything newer has /var and /var/lib with ownership of clamav / 200, when that should only apply from /var/lib/clamav as per the Dockerfile.

Originally we used COPY --link until realizing the UID/GID value mapping was not reliable, and that the clamav user and group could not be used with --chown with --link, so we created the user explicitly before installing a package that would create a clamav user/group, and reference that stable UID for --chown: https://github.com/moby/buildkit/issues/2987#issuecomment-1396753289

Reproduce the configuration in the reported Dockerfile with the docker-container and native docker drivers, using the listed buildx commands, and compare ownership under /var, /var/lib, and /var/lib/clamav. Trace the docker-container driver and its COPY --link --chown handling; done means only /var/lib/clamav has the requested ownership, with a regression test covering the difference.