`docker buildx build` hangs when using some cached layers
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 10.3k
- Forks
- 1.5k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 48
Description
Copypaste of https://github.com/moby/buildkit/issues/2009 with different versions and backtraces.
docker-20.10.7, buildx-github.com/docker/buildx v0.5.1 11057da37336192bfc57d81e02359ba7ba848e4a (nigthly) setup via docker buildx create --name builder-builder --platform arm64,amd64 --driver docker-container --driver-opt image=moby/buildkit:nightly --buildkitd-flags '--debug' --use.
Unfortunately cannot provide a reproducer, since it's internal project.
Logs after SIGQUIT to buildkitd: https://gist.github.com/a6a9de6c613949cf6bbababef2163b7d
In Dockerfile there is
COPY --from=cloner REVISION /REVISION
ENV RAILS_SERVE_STATIC_FILES=true
ENV PORT=3002
and in the docker buildx build output last lines are
=> [linux/amd64 scheduler 12/13] COPY --from=cloner /app /app 12.4s
=> [linux/arm64 scheduler 12/13] COPY --from=cloner /app /app 10.8s
=> [linux/amd64 scheduler 13/13] COPY --from=cloner REVISION /REVISION 5.4s
=> [linux/arm64 scheduler 13/13] COPY --from=cloner REVISION /REVISION 5.4s
perf top -g -p ${buildkitd_pid} shows
Children Self Shared Object Symbol
+ 99,78% 2,81% buildkitd [.] io.init
+ 99,71% 7,10% buildkitd [.] github.com/containerd/containerd/api/services/namespaces/v1.(*GetNamespaceResponse).Stri
+ 99,71% 2,87% buildkitd [.] github.com/containerd/containerd/api/services/namespaces/v1.(*ListNamespacesRequest).Str
+ 26,38% 0,00% buildkitd [.] github.com/gogo/protobuf/types.(*Syntax).EnumDescriptor
+ 26,38% 0,00% buildkitd [.] github.com/gogo/protobuf/types.init.19
+ 26,38% 0,00% buildkitd [.] github.com/gogo/protobuf/types.(*EnumValue).Unmarshal
+ 26,38% 0,00% buildkitd [.] github.com/moby/buildkit/cache/metadata.(*StorageItem).SetValue
+ 26,38% 0,00% buildkitd [.] github.com/moby/buildkit/util/grpcerrors.each
+ 26,38% 0,00% buildkitd [.] github.com/moby/buildkit/util/grpcerrors.FromGRPC
+ 26,38% 0,00% buildkitd [.] type.*+0x1f52a9
+ 26,38% 0,00% buildkitd [.] github.com/gogo/googleapis/google/rpc.(*DebugInfo).Equal
+ 26,38% 0,00% buildkitd [.] github.com/moby/buildkit/client/llb.(*State).Output
+ 26,38% 0,00% buildkitd [.] github.com/moby/buildkit/cache.migrateChainID
+ 26,38% 0,00% buildkitd [.] github.com/containerd/containerd/runtime/v2/runc/options.(*CheckpointOptions).XXX_Merge
+ 26,38% 0,00% buildkitd [.] github.com/containerd/containerd/plugin.Register
+ 26,38% 0,00% buildkitd [.] github.com/google/uuid.Parse
+ 26,38% 0,00% buildkitd [.] github.com/containerd/containerd/api/services/snapshots/v1.(*UpdateSnapshotRequest).Stri
+ 26,38% 0,00% buildkitd [.] github.com/containerd/containerd/api/services/namespaces/v1.(*ListNamespacesResponse).Ma
+ 26,38% 0,00% buildkitd [.] github.com/containerd/containerd/api/services/namespaces/v1.(*ListNamespacesRequest).Mar
+ 17,07% 16,78% buildkitd [.] runtime.mapaccess1_fast32
+ 14,52% 0,32% buildkitd [.] internal/reflectlite.maplen
+ 13,48% 1,11% buildkitd [.] github.com/containerd/containerd/api/services/namespaces/v1.(*GetNamespaceRequest).Unmar
+ 11,64% 11,41% buildkitd [.] runtime.typedslicecopy
+ 8,27% 8,16% buildkitd [.] runtime.findObject
+ 6,21% 6,10% buildkitd [.] type..eq.sync.poolLocal
+ 5,69% 5,60% buildkitd [.] runtime.mapaccess1_faststr
+ 5,56% 5,44% buildkitd [.] runtime.mapassign_fast32
+ 3,94% 3,88% buildkitd [.] reflect.typedmemclrpartial
+ 3,94% 3,89% buildkitd [.] runtime.mapaccess2_fast32
+ 2,79% 2,76% buildkitd [.] internal/reflectlite.typedmemmove
+ 2,67% 2,63% buildkitd [.] reflect.mapassign
+ 2,46% 2,41% buildkitd [.] reflect.typedslicecopy
+ 2,28% 2,26% buildkitd [.] type..eq.sync.Cond
100% reproducible with --cache-from, works fine without --cache-from.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked BuildKit issue, the supplied buildkitd SIGQUIT backtrace, and the perf output, then compare the build with and without --cache-from. No source file or test is named, and there is no public reproducer; done would require identifying the cache-related hang and verifying the behavior no longer occurs in the reported multi-platform build.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go
- Domain
- build-system, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100