docker multi-stage build; copy from builder stage statically linked binary some how alters resultant file to dynamically linked binary...

Description

Attempting to build a container from a multi-stage build which uses two stages; one builds a go binary and the other copies that single go-built static binary to itself.

When attempting to run the copioed binary, it fails to run. The error indicates it's a dynamically created binary. However, when digging into this, the binary on the base image is in fact statically built and the binary on the dest (final) container is dynamically linked. Use of strings shows that this is in fact the same application.

Other troubleshooting steps taken:

1. copied resultant binary from both images and compared them. The md5sum of the files differ from each other
2. Verified that the binary created in the build stage image is not a symlink or hard link.
3. Updated the build to directly output the resultant built binary to / from which the final stage copies yet the problem continues
4. running docker run -it <builder image id> vault-env results in expected behavior

Steps to reproduce the issue:

1. docker build --no-cache -t tool:dev -f docker/Dockerfile . using the Dockerfile below.
2. docker run -it tool:dev vault-env

Describe the results you received:

Run of the final container:

$ docker run -it tool:dev /tool/vault-env
/tool/vault-env: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /tool/vault-env)

from the builder container image:

ldd of builder image vault-env

root@8819ae8ea130:/# ldd vault-env 
        not a dynamic executable

md5sum of same

# md5sum vault-env 
53d33010f6baa32466d531acbdc257b9  vault-env

from the final container image:

ldd of final image vault-env

root@e14ccf3535f6:/tool# ldd vault-env 
./vault-env: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./vault-env)
        linux-vdso.so.1 (0x00007fff08ffc000)
        libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f3419bf0000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f34199fe000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f3419c18000)

md5sum of same

# md5sum vault-env 
501541e3fe10dfb096442f791bb36e37  vault-env

Describe the results you expected:

FATA[0000] no command is given, vault-env can't determine the entrypoint (command), please specify it explicitly or let the webhook query it (see documentation)  app=vault-env

Which is an error from the vault-env binary. The error is expected, of course.

Additional information you deem important (e.g. issue happens only occasionally):

The Dockerfile used:

FROM ubuntu:latest AS builder

ENV DEBIAN_FRONTEND=noninteractive

WORKDIR /

RUN apt update && \
    apt upgrade -y && \
    apt install -y golang git wget curl jq && \
    wget https://github.com/banzaicloud/bank-vaults/archive/refs/tags/1.13.0.tar.gz && \
    tar xfz 1.13.0.tar.gz && \
    cd bank-vaults-1.13.0/cmd/vault-env && \
    env GOARCH=amd64 GOOS=linux CGO_ENABLED=0 go build -a -installsuffix cgo -o /vault-env .


FROM ubuntu:latest

ENV PATH=$PATH:/usr/local/bin
ENV DEBIAN_FRONTEND=noninteractive
ENV VAULT_ADDR='https://192.168.7.208:8200' \
    VAULT_SKIP_VERIFY=true \
    VAULT_TOKEN='<REDACTED>' \
    VAULT_ROLE=default
ENV TOKEN=vault:secret/data/sre/tool#token

RUN apt update && \
    apt upgrade -y && \
    apt install -y bash ruby ruby-dev build-essential git curl jq && \
    gem install slack-ruby-client --no-doc && \
    echo 'y' | sh -c "$(curl -# -sL https://raw.githubusercontent.com/martindstone/pagerduty-cli/master/install.sh)" && \
    apt remove -y --purge build-essential && \
    apt autoremove -y && \
    rm -rf /var/lib/apt/lists/* || true

COPY templates/ /tool/templates/
COPY bin/ /tool/
COPY --from=builder /vault-env /tool/bin/

WORKDIR /tool

CMD ["./vault-env","tool-cli","-D","-n"]

Output of docker version:

Client: Docker Engine - Community
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        f0df350
 Built:             Wed Jun  2 11:56:41 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.7
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       b0f5bc3
  Built:            Wed Jun  2 11:54:53 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.6
  GitCommit:        d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Output of docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Build with BuildKit (Docker Inc., v0.5.1-docker)
  scan: Docker Scan (Docker Inc., v0.8.0)

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 88
 Server Version: 20.10.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d71fcd7d8303cbf684402823e425e9dd2e99285d
 runc version: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
  WARNING: You're not using the default seccomp profile
   Profile: /etc/docker/seccomp.json
 Kernel Version: 5.8.0-59-generic
 Operating System: Ubuntu 20.10
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 2.916GiB
 Name: jimconn-vmlinux
 ID: MVGE:SAUU:SXF6:AOTT:J573:X3ON:2IT2:45JS:UTFT:YAEX:MDD3:A42J
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: njbluescape
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

I've tested this in two places. My main working environment is a virtualbox Linux Ubuntu 20.10 distro and this is where I first encountered this issue. In an effort to troubleshoot this issue, I attempted to build and test this on my mac (Catalina) using the latest stable docker version, which results in the exact same issue.

Start by reproducing the report with the Dockerfile shown, using docker build --no-cache -t tool:dev -f docker/Dockerfile . and docker run -it tool:dev /tool/vault-env. Compare /vault-env in the builder stage with /tool/vault-env in the final image using ldd and md5sum; done means the multi-stage copy behavior is explained and the reported runtime result is corrected or documented.