moby / moby/buildkit

`docker buildx build` hangs when using some cached layers

Open
#2,275 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status/needs-more-info
Dominant language
Go
Stars
10.3k
Forks
1.5k
Avg merge
1d 23h
Merged PRs (30d)
48

Description

Copypaste of https://github.com/moby/buildkit/issues/2009 with different versions and backtraces.

docker-20.10.7, buildx-github.com/docker/buildx v0.5.1 11057da37336192bfc57d81e02359ba7ba848e4a (nigthly) setup via docker buildx create --name builder-builder --platform arm64,amd64 --driver docker-container --driver-opt image=moby/buildkit:nightly --buildkitd-flags '--debug' --use.

Unfortunately cannot provide a reproducer, since it's internal project.

Logs after SIGQUIT to buildkitd: https://gist.github.com/a6a9de6c613949cf6bbababef2163b7d
In Dockerfile there is

COPY --from=cloner REVISION /REVISION

ENV RAILS_SERVE_STATIC_FILES=true
ENV PORT=3002

and in the docker buildx build output last lines are

 => [linux/amd64 scheduler 12/13] COPY --from=cloner /app /app                                                                12.4s
 => [linux/arm64 scheduler 12/13] COPY --from=cloner /app /app                                                                10.8s
 => [linux/amd64 scheduler 13/13] COPY --from=cloner REVISION /REVISION                                                        5.4s
 => [linux/arm64 scheduler 13/13] COPY --from=cloner REVISION /REVISION                                                        5.4s

perf top -g -p ${buildkitd_pid} shows

  Children      Self  Shared Object     Symbol                                                                                      
+   99,78%     2,81%  buildkitd         [.] io.init
+   99,71%     7,10%  buildkitd         [.] github.com/containerd/containerd/api/services/namespaces/v1.(*GetNamespaceResponse).Stri
+   99,71%     2,87%  buildkitd         [.] github.com/containerd/containerd/api/services/namespaces/v1.(*ListNamespacesRequest).Str
+   26,38%     0,00%  buildkitd         [.] github.com/gogo/protobuf/types.(*Syntax).EnumDescriptor
+   26,38%     0,00%  buildkitd         [.] github.com/gogo/protobuf/types.init.19
+   26,38%     0,00%  buildkitd         [.] github.com/gogo/protobuf/types.(*EnumValue).Unmarshal
+   26,38%     0,00%  buildkitd         [.] github.com/moby/buildkit/cache/metadata.(*StorageItem).SetValue
+   26,38%     0,00%  buildkitd         [.] github.com/moby/buildkit/util/grpcerrors.each
+   26,38%     0,00%  buildkitd         [.] github.com/moby/buildkit/util/grpcerrors.FromGRPC
+   26,38%     0,00%  buildkitd         [.] type.*+0x1f52a9
+   26,38%     0,00%  buildkitd         [.] github.com/gogo/googleapis/google/rpc.(*DebugInfo).Equal
+   26,38%     0,00%  buildkitd         [.] github.com/moby/buildkit/client/llb.(*State).Output
+   26,38%     0,00%  buildkitd         [.] github.com/moby/buildkit/cache.migrateChainID
+   26,38%     0,00%  buildkitd         [.] github.com/containerd/containerd/runtime/v2/runc/options.(*CheckpointOptions).XXX_Merge
+   26,38%     0,00%  buildkitd         [.] github.com/containerd/containerd/plugin.Register
+   26,38%     0,00%  buildkitd         [.] github.com/google/uuid.Parse
+   26,38%     0,00%  buildkitd         [.] github.com/containerd/containerd/api/services/snapshots/v1.(*UpdateSnapshotRequest).Stri
+   26,38%     0,00%  buildkitd         [.] github.com/containerd/containerd/api/services/namespaces/v1.(*ListNamespacesResponse).Ma
+   26,38%     0,00%  buildkitd         [.] github.com/containerd/containerd/api/services/namespaces/v1.(*ListNamespacesRequest).Mar
+   17,07%    16,78%  buildkitd         [.] runtime.mapaccess1_fast32
+   14,52%     0,32%  buildkitd         [.] internal/reflectlite.maplen
+   13,48%     1,11%  buildkitd         [.] github.com/containerd/containerd/api/services/namespaces/v1.(*GetNamespaceRequest).Unmar
+   11,64%    11,41%  buildkitd         [.] runtime.typedslicecopy
+    8,27%     8,16%  buildkitd         [.] runtime.findObject
+    6,21%     6,10%  buildkitd         [.] type..eq.sync.poolLocal
+    5,69%     5,60%  buildkitd         [.] runtime.mapaccess1_faststr
+    5,56%     5,44%  buildkitd         [.] runtime.mapassign_fast32
+    3,94%     3,88%  buildkitd         [.] reflect.typedmemclrpartial
+    3,94%     3,89%  buildkitd         [.] runtime.mapaccess2_fast32
+    2,79%     2,76%  buildkitd         [.] internal/reflectlite.typedmemmove
+    2,67%     2,63%  buildkitd         [.] reflect.mapassign
+    2,46%     2,41%  buildkitd         [.] reflect.typedslicecopy
+    2,28%     2,26%  buildkitd         [.] type..eq.sync.Cond

100% reproducible with --cache-from, works fine without --cache-from.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked BuildKit issue, the supplied buildkitd SIGQUIT backtrace, and the perf output, then compare the build with and without --cache-from. No source file or test is named, and there is no public reproducer; done would require identifying the cache-related hang and verifying the behavior no longer occurs in the reported multi-platform build.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go
Domain
build-system, devops
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.