docker build a simple COPY of a 10 byte file takes over 2 minutes

Description

recently I have noticing that my container builds are running slowly. first i thought it was because of the SNAP store based installation, and i finally uninstalled it, that story here https://bugs.launchpad.net/snapd/+bug/1991475

so because of this i have been running all sort of unix monitoring tools to see if there is IO wait issues disk write issues and I didn't notice anything out of the ordinary .

I even went to the extent by providing it cpu and memory units thinking that will help,. I don't know if i use those params right but here it what it typed

ocker build -f Dockerfile -t  fobar:image -m=16g --cpuset-cpus=3 -c=2048 .

and I had the kernel log on tail in one of the screen sessions and i noticed this

Oct 07 00:18:11 nayana-295 kernel: INFO: task dockerd:46662 blocked for more than 120 seconds.
Oct 07 00:18:11 nayana-295 kernel:       Not tainted 5.15.0-48-generic #54-Ubuntu
Oct 07 00:18:11 nayana-295 kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
Oct 07 00:18:11 nayana-295 kernel: task:dockerd         state:D stack:    0 pid:46662 ppid:     1 flags:0x00004002
Oct 07 00:18:11 nayana-295 kernel: Call Trace:
Oct 07 00:18:11 nayana-295 kernel:  <TASK>
Oct 07 00:18:11 nayana-295 kernel:  __schedule+0x23d/0x5a0
Oct 07 00:18:11 nayana-295 kernel:  schedule+0x4e/0xc0
Oct 07 00:18:11 nayana-295 kernel:  wb_wait_for_completion+0x59/0x90
Oct 07 00:18:11 nayana-295 kernel:  ? wait_woken+0x70/0x70
Oct 07 00:18:11 nayana-295 kernel:  sync_inodes_sb+0xbe/0x100
Oct 07 00:18:11 nayana-295 kernel:  sync_filesystem+0x6a/0xb0
Oct 07 00:18:11 nayana-295 kernel:  ovl_sync_fs+0x61/0xa0 [overlay]
Oct 07 00:18:11 nayana-295 kernel:  sync_filesystem+0x84/0xb0
Oct 07 00:18:11 nayana-295 kernel:  generic_shutdown_super+0x27/0x120
Oct 07 00:18:11 nayana-295 kernel:  kill_anon_super+0x18/0x30
Oct 07 00:18:11 nayana-295 kernel:  deactivate_locked_super+0x3b/0xb0
Oct 07 00:18:11 nayana-295 kernel:  deactivate_super+0x44/0x50
Oct 07 00:18:11 nayana-295 kernel:  cleanup_mnt+0x150/0x1d0
Oct 07 00:18:11 nayana-295 kernel:  ? mntput_no_expire+0x19c/0x2d0
Oct 07 00:18:11 nayana-295 kernel:  __cleanup_mnt+0x12/0x20
Oct 07 00:18:11 nayana-295 kernel:  task_work_run+0x6d/0xb0
Oct 07 00:18:11 nayana-295 kernel:  exit_to_user_mode_loop+0x157/0x160
Oct 07 00:18:11 nayana-295 kernel:  exit_to_user_mode_prepare+0xa0/0xb0
Oct 07 00:18:11 nayana-295 kernel:  syscall_exit_to_user_mode+0x27/0x50
Oct 07 00:18:11 nayana-295 kernel:  do_syscall_64+0x69/0xc0
Oct 07 00:18:11 nayana-295 kernel:  ? exit_to_user_mode_prepare+0x37/0xb0
Oct 07 00:18:11 nayana-295 kernel:  ? syscall_exit_to_user_mode+0x27/0x50
Oct 07 00:18:11 nayana-295 kernel:  ? __x64_sys_close+0x11/0x50
Oct 07 00:18:11 nayana-295 kernel:  ? do_syscall_64+0x69/0xc0
Oct 07 00:18:11 nayana-295 kernel:  ? do_syscall_64+0x69/0xc0
Oct 07 00:18:11 nayana-295 kernel:  ? exit_to_user_mode_prepare+0x37/0xb0
Oct 07 00:18:11 nayana-295 kernel:  ? syscall_exit_to_user_mode+0x27/0x50
ct 07 00:18:11 nayana-295 kernel:  ? syscall_exit_to_user_mode+0x27/0x50
Oct 07 00:18:11 nayana-295 kernel:  ? __x64_sys_read+0x19/0x20
Oct 07 00:18:11 nayana-295 kernel:  ? do_syscall_64+0x69/0xc0
Oct 07 00:18:11 nayana-295 kernel:  ? do_syscall_64+0x69/0xc0
Oct 07 00:18:11 nayana-295 kernel:  ? do_syscall_64+0x69/0xc0
Oct 07 00:18:11 nayana-295 kernel:  ? do_syscall_64+0x69/0xc0
Oct 07 00:18:11 nayana-295 kernel:  ? sysvec_reschedule_ipi+0x78/0xe0
Oct 07 00:18:11 nayana-295 kernel:  entry_SYSCALL_64_after_hwframe+0x61/0xcb
Oct 07 00:18:11 nayana-295 kernel: RIP: 0033:0x55b4c8a5109b
Oct 07 00:18:11 nayana-295 kernel: RSP: 002b:000000c0006dc330 EFLAGS: 00000206 ORIG_RAX: 00000000000000a6
Oct 07 00:18:11 nayana-295 kernel: RAX: 0000000000000000 RBX: 000000c000063000 RCX: 000055b4c8a5109b
Oct 07 00:18:11 nayana-295 kernel: RDX: 0000000000000000 RSI: 0000000000000002 RDI: 000000c0019a38f0
Oct 07 00:18:11 nayana-295 kernel: RBP: 000000c0006dc390 R08: 00007f0b1feb3a01 R09: 0000000000000000
Oct 07 00:18:11 nayana-295 kernel: R10: 00007f0af00b9748 R11: 0000000000000206 R12: 000000c0019a38f0
Oct 07 00:18:11 nayana-295 kernel: R13: 0000000000000001 R14: 000000c000669d40 R15: ffffffffffffffff
Oct 07 00:18:11 nayana-295 kernel:  </TASK>

the version of docker i am using

Docker version 20.10.12, build 20.10.12-0ubuntu4

I don't know what to make out of it, because i am still struggling to figure out how to build the docker cli from source, that's a different story and not related to this report

should i try getting rid of some of the left over intermediary images? or clear everything out?

Reproduce

simply create a docker file with a simple COPY instruction for a file and observe the time it takes to finish that instruction

Expected behavior

COPY .env ./.env should finish up instantly in my opinion

docker version

Client:
 Version:           20.10.12
 API version:       1.41
 Go version:        go1.17.3
 Git commit:        20.10.12-0ubuntu4
 Built:             Mon Mar  7 17:10:06 2022
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.12
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.17.3
  Git commit:       20.10.12-0ubuntu4
  Built:            Mon Mar  7 15:57:50 2022
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.5.9-0ubuntu3
  GitCommit:        
 runc:
  Version:          1.1.0-0ubuntu1
  GitCommit:        
 docker-init:
  Version:          0.19.0
  GitCommit:

docker info

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 23
  Running: 0
  Paused: 0
  Stopped: 23
 Images: 85
 Server Version: 20.10.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 
 runc version: 
 init version: 
 Security Options:
  apparmor
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.15.0-48-generic
 Operating System: Ubuntu 22.04.1 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.27GiB
 Name: nayana-295
 ID: KA2L:GCBL:BE4O:ULHG:65L4:CDK5:IFXZ:JS4Y:6JTY:S3HF:6CVM:ZB5P
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional Info

if you can help me out on how to build the cli from source I would be able to capture further details on this issue

right now i am stuck here

i have tried

make dev
make vendor
./scripts/vendor init
./scripts/vendor update
make binary
./scripts/binary

they all result in the same error as below, i thought it was because i was using go19, so i tried with go18 also same problem

./scripts/build/binary
Building static docker-linux-amd64
+ go build -o build/docker-linux-amd64 -tags  osusergo pkcs11 -ldflags  -X "github.com/docker/cli/cli/version.GitCommit=d0bd373986" -X "github.com/docker/cli/cli/version.BuildTime=2022-10-06T17:30:59Z" -X "github.com/docker/cli/cli/version.Version=22.06.0-beta.0-167-gd0bd373986.m" -extldflags -static -buildmode=pie github.com/docker/cli/cmd/docker
go: inconsistent vendoring in /home/nayana/code/mine/docker-cli:
        github.com/creack/pty@v1.1.18: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/go-logr/logr@v1.2.3: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/go-logr/stdr@v1.2.2: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/google/go-cmp@v0.5.8: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        go.opentelemetry.io/otel@v1.10.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        go.opentelemetry.io/otel/metric@v0.32.1: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        go.opentelemetry.io/otel/trace@v1.10.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt
        github.com/Azure/go-ansiterm@v0.0.0-20210617225240-d185dfc1b5a1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/Microsoft/go-winio@v0.5.2: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/beorn7/perks@v1.0.1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/cespare/xxhash/v2@v2.1.2: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/containerd/containerd@v1.6.8: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/creack/pty@v1.1.11: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/docker/distribution@v2.8.1+incompatible: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/docker/docker-credential-helpers@v0.7.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/docker/go@v1.5.1-1.0.20160303222718-d30aec9fd63c: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/docker/go-connections@v0.4.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/docker/go-events@v0.0.0-20190806004212-e31b211e4f1c: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/docker/go-metrics@v0.0.1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/docker/go-units@v0.5.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/fvbommel/sortorder@v1.0.2: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/gogo/protobuf@v1.3.2: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/golang/protobuf@v1.5.2: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/google/go-cmp@v0.5.7: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/google/shlex@v0.0.0-20191202100458-e7afc7fbc510: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/gorilla/mux@v1.8.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/imdario/mergo@v0.3.12: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/inconshreveable/mousetrap@v1.0.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/klauspost/compress@v1.15.9: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/mattn/go-runewidth@v0.0.13: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/matttproud/golang_protobuf_extensions@v1.0.2-0.20181231171920-c182affec369: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/miekg/pkcs11@v1.1.1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/mitchellh/mapstructure@v1.3.2: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/moby/buildkit@v0.10.4: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/moby/swarmkit/v2@v2.0.0-20220721174824-48dd89375d0a: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/moby/sys/sequential@v0.0.0-20220829095930-b22ba8a69b30: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/moby/sys/signal@v0.7.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/moby/sys/symlink@v0.2.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/moby/term@v0.0.0-20210619224110-3f7ff695adc6: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/morikuni/aec@v1.0.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/opencontainers/go-digest@v1.0.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/opencontainers/image-spec@v1.0.3-0.20220303224323-02efb9a75ee1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/opencontainers/runc@v1.1.2: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/pkg/errors@v0.9.1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/prometheus/client_golang@v1.12.1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/prometheus/client_model@v0.2.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/prometheus/common@v0.32.1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/prometheus/procfs@v0.7.3: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/rivo/uniseg@v0.2.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/sirupsen/logrus@v1.8.1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/spf13/cobra@v1.5.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/spf13/pflag@v1.0.5: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/theupdateframework/notary@v0.7.1-0.20210315103452-bf96a202a09a: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/tonistiigi/go-rosetta@v0.0.0-20200727161949-f79598599c5d: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/xeipuuv/gojsonpointer@v0.0.0-20190905194746-02993c407bfb: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/xeipuuv/gojsonreference@v0.0.0-20180127040603-bd5ef7bd5415: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/xeipuuv/gojsonschema@v1.2.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        go.etcd.io/etcd/raft/v3@v3.5.2: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/crypto@v0.0.0-20220315160706-3147a52a75dd: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/net@v0.0.0-20220906165146-f3363e06e74c: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/sys@v0.0.0-20220825204002-c680a09ffe64: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/term@v0.0.0-20210927222741-03fcf44c2211: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/text@v0.3.7: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/time@v0.0.0-20210723032227-1f47c861a9ac: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        golang.org/x/xerrors@v0.0.0-20200804184101-5ec99f83aff1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        google.golang.org/genproto@v0.0.0-20211208223120-3a66f561d7aa: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        google.golang.org/grpc@v1.45.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        google.golang.org/protobuf@v1.27.1: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        gopkg.in/yaml.v2@v2.4.0: is marked as explicit in vendor/modules.txt, but not explicitly required in go.mod
        github.com/docker/docker@v20.10.18+incompatible: is marked as replaced in vendor/modules.txt, but not replaced in go.mod
        github.com/docker/docker: is marked as replaced in vendor/modules.txt, but not replaced in go.mod
        github.com/google/certificate-transparency-go: is marked as replaced in vendor/modules.txt, but not replaced in go.mod

        To ignore the vendor directory, use -mod=readonly or -mod=mod.
        To sync the vendor directory, run:
                go mod vendor

Start by reproducing the reported delay with a minimal Dockerfile containing a simple COPY instruction, using the Docker version, storage driver, and kernel details provided. Review the build path and the mentioned scripts/build/binary entry point, then compare behavior with a current build; done means identifying a reproducible CLI-side cause or documenting that the issue belongs elsewhere.