Failure to push when using multi-node context
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 4.5k
- Forks
- 682
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 29
Description
I will try to describe my issue as good as I can... much of the setup is done by trail and error, so I could just be doing it wrong, but the documentation for multi-node buildx instances is quite sparse at the moment, so it's the best I could!
Scenario:
I have connected a ARM64 machines docker daemon to one of my AMD64 machines through buildx to be able to utilize the native ARM64 engine for ARM64 builds. While the engines seems to work great together while building, when they are about to push they stop after the AMD64 machine have pushed the first tag.
There are no errors in the build logs, so it looks like it was built and pushed successfully while no images are pushed to the registries.
I have tried both by connecting to the remote daemon over tcp and over ssh, while none of them seems to work for me.
Connection was done through docker buildx create --append --name <builder-name> tcp://ip:2375
It does though, seem like the registries do notice that something was being pushed as they say that the last update was "1 minute ago", while no images have been changed.
The last part of the build log looks like this when it fails to push successfully:
#11 exporting to image
#11 exporting layers
#11 exporting layers 15.1s done
#11 pushing layers
#11 exporting manifest sha256:e79069017c796a48ecd0122aa6bb4b5a71d5366dad15a22d8dae3cf3445ffe85 0.0s done
#11 exporting config sha256:7539d58986be67c54043cf7c10f759f17d9f2c27cf48baa904ce6601e915af89 0.0s done
#11 exporting manifest list sha256:c13816f3f05006682736fcba1eb7cc9e38fd972c31d3a26155d8a80a33de1539 0.0s done
#11 pushing layers 6.6s done
#11 pushing manifest for registry.gitlab.com/jitesoft/dockerfiles/node-base
#11 pushing manifest for registry.gitlab.com/jitesoft/dockerfiles/node-base 1.6s done
#11 pushing layers 5.0s done
#11 pushing manifest for docker.io/jitesoft/node-base
#11 pushing manifest for docker.io/jitesoft/node-base 0.8s done
#11 DONE 29.2s
#16 merging manifest list registry.gitlab.com/jitesoft/dockerfiles/node-base...
#16 DONE 4.4s
While a "real" successful one looks like the following (using only the AMD64 machine with Qemu):
#13 exporting to image
#13 exporting layers
#13 exporting layers 12.0s done
#13 exporting manifest sha256:120e8ce6e70745087f148f6991b9c072a331a42b15224d7b073382dbd7f8862f 0.0s done
#13 exporting config sha256:708ed343e10cf46f540039b355a21fa962be06dfb0be6adf88385e11974872fb 0.0s done
#13 exporting manifest sha256:0a495831b15cdf834d3d5fbd56acb2834a9161d42fac9fe8ec69c2435807a79f 0.0s done
#13 exporting config sha256:33d5f0ebafc4f4a6b33acd9726a2517e916713616c539f0c28c210976f84b424 0.0s done
#13 exporting manifest list sha256:c5a0e0562036da3573f03c609923266779df8dc4b2f3d5de6fa1d4b11aec108a 0.0s done
#13 pushing layers
#13 pushing layers 6.0s done
#13 pushing manifest for registry.gitlab.com/jitesoft/dockerfiles/node-base:12.13.0-slim
#13 pushing manifest for registry.gitlab.com/jitesoft/dockerfiles/node-base:12.13.0-slim 3.9s done
#13 pushing layers 4.9s done
#13 pushing manifest for docker.io/jitesoft/node-base:12.13.0-slim
#13 pushing manifest for docker.io/jitesoft/node-base:12.13.0-slim 1.5s done
#13 pushing layers 1.8s done
#13 pushing manifest for registry.gitlab.com/jitesoft/dockerfiles/node-base:12-slim
#13 pushing manifest for registry.gitlab.com/jitesoft/dockerfiles/node-base:12-slim 3.0s done
#13 pushing layers 1.4s done
#13 pushing manifest for docker.io/jitesoft/node-base:12-slim
#13 pushing manifest for docker.io/jitesoft/node-base:12-slim 1.1s done
#13 pushing layers 1.8s done
#13 pushing manifest for registry.gitlab.com/jitesoft/dockerfiles/node-base:stable-slim
#13 pushing manifest for registry.gitlab.com/jitesoft/dockerfiles/node-base:stable-slim 3.2s done
#13 pushing layers 1.0s done
#13 pushing manifest for docker.io/jitesoft/node-base:stable-slim
#13 pushing manifest for docker.io/jitesoft/node-base:stable-slim 1.4s done
#13 pushing layers 1.9s done
#13 pushing manifest for registry.gitlab.com/jitesoft/dockerfiles/node-base:lts-slim
#13 pushing manifest for registry.gitlab.com/jitesoft/dockerfiles/node-base:lts-slim 3.5s done
#13 pushing layers 1.1s done
#13 pushing manifest for docker.io/jitesoft/node-base:lts-slim
#13 pushing manifest for docker.io/jitesoft/node-base:lts-slim 1.1s done
#13 pushing layers 2.0s done
#13 pushing manifest for registry.gitlab.com/jitesoft/dockerfiles/node-base:erbium-slim
#13 pushing layers
#13 pushing manifest for registry.gitlab.com/jitesoft/dockerfiles/node-base:erbium-slim 8.1s done
#13 pushing layers 1.0s done
#13 pushing manifest for docker.io/jitesoft/node-base:erbium-slim
#13 pushing manifest for docker.io/jitesoft/node-base:erbium-slim 1.0s done
#13 DONE 62.8s
If there are any logs that I can't find that would help in debugging the issue or any more information that you could need, please let me know!
Docker version output:
AMD machine:
Client: Docker Engine - Community Version: 19.03.4 API version: 1.40 Go version: go1.12.10 Git commit: 9013bf583a Built: Fri Oct 18 15:54:09 2019 OS/Arch: linux/amd64 Experimental: true Server: Docker Engine - Community Engine: Version: 19.03.4 API version: 1.40 (minimum version 1.12) Go version: go1.12.10 Git commit: 9013bf583a Built: Fri Oct 18 15:52:40 2019 OS/Arch: linux/amd64 Experimental: true containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
ARM machine:
Client: Docker Engine - Community
Version: 19.03.4
API version: 1.40
Go version: go1.12.10
Git commit: 9013bf5
Built: Fri Oct 18 15:52:24 2019
OS/Arch: linux/arm64
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 19.03.4
API version: 1.40 (minimum version 1.12)
Go version: go1.12.10
Git commit: 9013bf5
Built: Fri Oct 18 15:50:53 2019
OS/Arch: linux/arm64
Experimental: true
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683
Buildx version:
AMD machine:
github.com/docker/buildx v0.3.1 6db68d029599c6710a32aa7adcba8e5a344795a7
ARM machine:
github.com/docker/buildx v0.3.1-tp-docker 6db68d029599c6710a32aa7adcba8e5a344795a7
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the multi-node build with docker buildx create --append over TCP or SSH, comparing its push output with the single-node QEMU output. Investigate the multi-node export and manifest-list push path; done means all expected tags and images appear in both registries, with diagnostic logs explaining any failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- build-system, devops, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100