github action cache error when pushing cache

Contributing guidelines

• I've read the contributing guidelines and wholeheartedly agree

I've found a bug and checked that ...

• ... the documentation does not mention anything about my problem
• ... there are no open or closed issues that are related to my problem

Description

When building images and pushing the build cache to github actions cache in mode=max, there are many times similar but different errors:

1. Error: buildx failed with: ERROR: failed to solve: error writing layer blob: Patch "https://artifactcache.actions.githubusercontent.com/..../_apis/artifactcache/caches/11111": read tcp 172.17.0.2:40814->142.250.141.82:443: use of closed network connection seems to come from #367
2. Gets stuck in exporting layer

And trust me, it really gets stuck:

Expected behaviour

There shouldn't be any timeouts or reuse of a closed connection.

Actual behaviour

It works unpredictably

Buildx version

github.com/docker/buildx 0.10.4+azure-1 c513d34049e499c53468deac6c4267ee72948f02

Docker info

Client:
   Context:    default
   Debug Mode: false
   Plugins:
    buildx: Docker Buildx (Docker Inc., 0.10.4+azure-1)
    compose: Docker Compose (Docker Inc., 2.17.2+azure-1)
  
  Server:
   Containers: 0
    Running: 0
    Paused: 0
    Stopped: 0
   Images: 18
   Server Version: 20.10.23+azure-2
   Storage Driver: overlay2
    Backing Filesystem: extfs
    Supports d_type: true
    Native Overlay Diff: false
    userxattr: false
   Logging Driver: json-file
   Cgroup Driver: cgroupfs
   Cgroup Version: 2
   Plugins:
    Volume: local
    Network: bridge host ipvlan macvlan null overlay
    Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
   Swarm: inactive
   Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
   Default Runtime: runc
   Init Binary: docker-init
   containerd version: 1e1ea6e986c6c86565bc33d52e34b81b3e2bc71f
   runc version: f19387a6bec4944c770f7668ab51c4348d9c2f38
   init version: 
   Security Options:
    apparmor
    seccomp
     Profile: default
    cgroupns
   Kernel Version: 5.15.0-1035-azure
   Operating System: Ubuntu 22.04.2 LTS
   OSType: linux
   Architecture: x86_64
   CPUs: 2
   Total Memory: 6.781GiB
   Name: fv-az251-622
   ID: 3Q5N:TB6N:KRBU:S5UK:CFVR:VASN:GQOQ:7YNN:ZAWD:NZJQ:ZO4T:DFAG
   Docker Root Dir: /var/lib/docker
   Debug Mode: false
   Username: githubactions
   Registry: https://index.docker.io/v1/
   Labels:
   Experimental: false
   Insecure Registries:
    127.0.0.0/8
   Live Restore Enabled: false

Builders list

The one created by the github action:

docker/setup-buildx-action@v2

Configuration

# Github actions yaml
      - name: Build docker dev
        uses: docker/build-push-action@v4
        with:
          # load: true makes the image available to docker compose
          load: true
          tags: |
            my-image:latest
          # temp workaround for newer buildx where images outputed by default are not cloud run compatible https://github.com/docker/buildx/issues/1533
          provenance: false
          cache-from: |
            type=gha,scope=my-image/${{ github.base_ref }}
            type=gha,scope=my-image/${{ github.ref_name }}
            type=gha,scope=my-image/dev-${{ github.base_ref }}
            type=gha,scope=my-image/dev-${{ github.ref_name }}
            type=registry,ref=registry.url/cache/my-image:latest
            type=registry,ref=registry.url/cache/my-image:newest
            type=registry,ref=registry.url/cache/my-image:dev-latest
            type=registry,ref=registry.url/cache/my-image:dev-newest
          cache-to: |
            type=gha,scope=my-image/dev-${{ github.ref_name }},mode=max
            type=registry,ref=registry.url/cache/my-image:dev-newest,mode=max

Build logs

#33 exporting to GitHub cache
#33 preparing build cache for export 6.1s done
#33 writing layer sha256:aaaaaaaaaaaaaaaaaaaaaaaaa 1.3s done
#33 writing layer sha256:bbbbbbbbbbbbbbbbbbbbbbbbb 1.2s done
#33 writing layer sha256:ccccccccccccccccccccccccc 0.6s done
#33 writing layer sha256:ddddddddddddddddddddddddd 2.6s done
#33 ERROR: error writing layer blob: Patch "https://artifactcache.actions.githubusercontent.com/.................../_apis/artifactcache/caches/13632": read tcp 172.17.0.2:36796->142.251.163.82:443: use of closed network connection

Additional info

It seems like most of the trouble is in github actions cache

The report names no source files or tests; start by reproducing the build with docker/setup-buildx-action@v2, docker/build-push-action@v4, and the shown type=gha cache-to configuration. Investigate the GitHub Actions cache exporter around the reported closed-connection and stuck-export symptoms. Done means cache export completes reliably without timeouts or reused closed connections.