`docker buildx build` hangs when using some cached layers

buildx-0.5.1, moby/buildkit:buildx-stable-1 (be8e8392f56c), Docker version 20.10.5, build 55c4c88 on linux x86_64.

Unfortunately cannot provide a reproducer since it's an internal project.

docker buildx build --cache-from … --cache-to type=inline --pull . hangs on various steps indefinitely (or for a very long time, in GH Actions there was 6 hour timeout). buildkit container is started with docker buildx create --name builder-builder --driver docker-container --driver-opt image=moby/buildkit:buildx-stable-1 --buildkitd-flags --debug --use (setup is stolen from docker/setup-buildx-action@v1 GH action), logs after sending SIGQUIT is here. This time the hang was on

 => CACHED [… 11/13] RUN bundle exec bootsnap precompile --gemfile config/ app/ lib/    282.6s

During the hang buildkitd eats some cpu. perf top -g shows

  Children      Self  Shared Object     Symbol
+   75,62%     0,39%  buildkitd         [.] github.com/moby/buildkit/solver.(*cacheManager).ensurePersistentKey
+   68,22%     0,08%  buildkitd         [.] github.com/moby/buildkit/solver/bboltcachestorage.(*Store).HasLink
+   67,33%     0,28%  buildkitd         [.] go.etcd.io/bbolt.(*DB).View
+   59,37%     0,34%  buildkitd         [.] github.com/moby/buildkit/solver/bboltcachestorage.(*Store).HasLink.func1
+   39,27%     0,44%  buildkitd         [.] go.etcd.io/bbolt.(*Cursor).seek
+   38,38%     9,83%  buildkitd         [.] go.etcd.io/bbolt.(*Cursor).search
+   29,51%     0,52%  buildkitd         [.] go.etcd.io/bbolt.(*Cursor).searchPage
+   24,31%     0,33%  buildkitd         [.] go.etcd.io/bbolt.(*Bucket).Bucket
+   22,37%     0,07%  buildkitd         [.] runtime.systemstack
+   21,22%     0,17%  buildkitd         [.] runtime.gcDrain
+   21,22%     0,24%  buildkitd         [.] go.etcd.io/bbolt.(*Bucket).Get
+   19,76%    10,83%  buildkitd         [.] runtime.scanobject
+   17,86%     2,92%  buildkitd         [.] sort.Search
+   15,52%     7,50%  buildkitd         [.] runtime.mallocgc
+   11,83%    11,26%  buildkitd         [.] cmpbody
+    9,81%     0,00%  buildkitd         [.] runtime.goexit
+    9,51%     0,00%  buildkitd         [.] github.com/moby/buildkit/solver/internal/pipe.NewWithFunction.func2
+    9,51%     0,00%  buildkitd         [.] github.com/moby/buildkit/solver.(*edge).loadCache-fm
+    9,51%     0,00%  buildkitd         [.] github.com/moby/buildkit/solver.(*edge).loadCache
+    9,51%     0,00%  buildkitd         [.] github.com/moby/buildkit/solver.(*sharedOp).LoadCache
+    9,51%     0,00%  buildkitd         [.] github.com/moby/buildkit/solver.(*combinedCacheManager).Load
+    9,51%     0,00%  buildkitd         [.] github.com/moby/buildkit/solver.(*cacheManager).Save
+    9,00%     1,36%  buildkitd         [.] runtime.growslice
+    8,40%     0,46%  buildkitd         [.] encoding/json.Marshal
+    8,24%     0,32%  buildkitd         [.] go.etcd.io/bbolt.(*Cursor).nsearch
+    7,29%     6,73%  buildkitd         [.] runtime.findObject
+    6,21%     0,79%  buildkitd         [.] go.etcd.io/bbolt.(*DB).beginTx
+    6,16%     0,02%  buildkitd         [.] go.etcd.io/bbolt.(*DB).Begin
+    5,88%     0,09%  buildkitd         [.] encoding/json.(*encodeState).marshal
+    5,69%     0,31%  buildkitd         [.] runtime.newobject
+    5,42%     0,24%  buildkitd         [.] encoding/json.(*encodeState).reflectValue
+    4,98%     1,01%  buildkitd         [.] go.etcd.io/bbolt.(*Bucket).openBucket
+    4,68%     4,52%  buildkitd         [.] runtime.heapBitsSetType
+    3,98%     0,01%  buildkitd         [.] encoding/json.structEncoder.encode-fm
+    3,84%     0,67%  buildkitd         [.] encoding/json.structEncoder.encode
+    3,46%     0,49%  buildkitd         [.] go.etcd.io/bbolt.(*Tx).init

I can reproduce it every time building the image and can provide any additional info if needed. If --cache-from is removed — image is built just fine.

Start with the --cache-from build command and compare it with the working build without that option. Investigate the perf-indicated paths solver.(*cacheManager).ensurePersistentKey, bboltcachestorage.Store.HasLink, and bbolt cursor operations, using the linked SIGQUIT logs; done means cached builds no longer hang or consume CPU indefinitely.