moby / moby/buildkit

COPY --exclude fails to parse parameter expansion correctly?

Open
#6,072 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status/triage
Dominant language
Go
Stars
10.3k
Forks
1.5k
Avg merge
1d 23h
Merged PRs (30d)
48

Description

Contributing guidelines and issue reporting guide
Well-formed report checklist
  • I have found a bug that the documentation does not mention anything about my problem
  • I have found a bug that there are no open or closed issues that are related to my problem
  • I have provided version/information about my environment and done my best to provide a reproducer
Description of bug

Bug description

I expected environment replacement to work with the COPY --exclude option; otherwise its difficult to copy all-but a few, that don't collectively match a simple * regex.
Currently, for all variations of quoting or re-formatting, I can't seem to get it to work. Bug?

My use case Ofc, I use .dockerignore to ignore files that should never be considered. Afterwards, I have files that change rarely, that impact later docker stages (ex, package manager dependency files, 3rd party libraries and their package manager dependencies), and then there are the "live" frequently changing files (but don't need a slow post-include build step). Ideally, I'd 1) COPY the minimum files needed for 2) Slow build steps (ex pkg installation), and lastly include/COPY everything else. (I already use multi-stage to minimize/separate (re-)compilations, and have started using the COPY --link).

Anyway, the below dockerfile is MRE for my case; I have a large list of files/directories I don't need earlier, out of a larger list of files/directories I do need earlier.
I'd expected this combo of features to just work tho, based off the docs.

Reproduction

mkdir -p /tmp/test_docker_copy_exclude && cd /tmp/test_docker_copy_exclude
for var in a b c d e f ; do echo $var > $var ; done; ls -al
DEMO_COPY_LATER=" d e f"
echo "${DEMO_COPY_LATER// / --exclude=}" . .

vim Dockerfile

# syntax=docker/dockerfile:1.7-labs
# check=error=true
FROM scratch
ARG COPY_LATER=" d e f"
# Note, same result with and without double quotes around the expansion
COPY "${COPY_LATER// / --exclude=}" . .
RUN cat ./* > x && echo "Do stuff with: " && ls -al
COPY $COPY_LATER .
RUN cat ./* > y && echo "Do stuff with: " && ls -al

docker build -t tmp_test_docker_copy_exclude .

[+] Building 0.2s (3/3) FINISHED                                                         docker:default
 => [internal] load build definition from Dockerfile                                               0.0s
 => => transferring dockerfile: 294B                                                               0.0s
 => resolve image config for docker-image://docker.io/docker/dockerfile:1.7-labs                   0.1s
 => CACHED docker-image://docker.io/docker/dockerfile:1.7-labs@sha256:b99fecfe00268a8b556fad7d9c3  0.0s
Dockerfile:5
--------------------
   3 |     FROM scratch
   4 |     ARG COPY_LATER=" d e f"
   5 | >>> COPY "${COPY_LATER// / --exclude=}" . .
   6 |     RUN cat ./* > x && echo "Do stuff with: " && ls -al
   7 |     COPY $COPY_LATER .
--------------------
ERROR: failed to solve: failed to process "\"${COPY_LATER//": syntax error: missing '/' in ${}

Version information

docker info docker info Client: Docker Engine - Community Version: 27.5.1 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.20.0 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.32.4 Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 1
Server Version: 27.5.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: false
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 nvidia runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bcc810d6b9066471b0b6fa75f557a15a1cbf31bb
runc version: v1.2.4-0-g6c52b3f
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
Kernel Version: 5.15.0-1089-azure
Operating System: Ubuntu 20.04.6 LTS
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 108.1GiB
Name:
ID:
Docker Root Dir: /mnt/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

bash --version GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by running the provided minimal Dockerfile reproduction and inspect BuildKit's Dockerfile parser handling of COPY --exclude with parameter expansion. Done means the documented environment replacement parses for this form and a regression test covers the reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
dockerfile, go
Domain
build-system, devtools
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.