Cache image layers extraction is very slow without any apparent reason, even on high end hardware

Start with the supplied Dockerfile and the buildctl-daemonless.sh command using registry mode=max cache export/import and rootless BuildKit v0.27.0. Reproduce the slow extraction, then trace the cache layer extraction path and compare its timing with recomputation. Done means the reported extraction slowdown is explained and addressed without regressing the reproduction.

Contributing guidelines and issue reporting guide

• I've read the contributing guidelines and wholeheartedly agree. I've also read the issue reporting guide.

Well-formed report checklist

• I have found a bug that the documentation does not mention anything about my problem
• I have found a bug that there are no open or closed issues that are related to my problem
• I have provided version/information about my environment and done my best to provide a reproducer

Bug description

Similar to #2874

I am running rootless buildkit for CI builds and i'm using the cache with mode=max to cache even intermediate layers and speedup builds.
However, even when running with high performance HW (lot of CPUs and RAM + local SSD), it takes a lot of time to extract layers

Sometime the extraction is so slow that recomputing the layer is actually faster

I don't understand why the extraction process is so slow, considering that both cpu and disk usage is very low during the extraction

Increasing or decreasing compression level does not seems to affect the extraction time

Reproduction

Dockerfile

FROM golang:1.25

COPY <<EOL go.mod
module mymodule

go 1.25

require (
	cloud.google.com/go/containeranalysis v0.14.2
	cloud.google.com/go/grafeas v0.3.17
	cloud.google.com/go/kms v1.25.0
	github.com/Masterminds/sprig/v3 v3.3.0
	github.com/bmatcuk/doublestar/v4 v4.10.0
	github.com/cockroachdb/errors v1.12.0
	github.com/go-git/go-git/v5 v5.16.4
	github.com/sirupsen/logrus v1.9.4
	github.com/slack-go/slack v0.17.3
	github.com/spf13/cobra v1.10.2
	github.com/spf13/pflag v1.0.10
	github.com/spf13/viper v1.21.0
	github.com/stretchr/testify v1.11.1
	golang.org/x/crypto v0.47.0
	golang.org/x/oauth2 v0.34.0
	golang.org/x/sync v0.19.0
	google.golang.org/api v0.263.0
	google.golang.org/genproto v0.0.0-20260128011058-8636f8732409
	google.golang.org/grpc v1.78.0
	google.golang.org/protobuf v1.36.11
	gopkg.in/yaml.v3 v3.0.1
)
EOL

RUN go mod download

RUN echo hello world

Spin up a registry to pull and push the cache

docker run --name registry -p 5000:5000 registry:2

In another terminal, run the first build

docker run --rm -it --privileged --entrypoint buildctl-daemonless.sh --network host -v $PWD:/home/user/files  moby/buildkit:v0.27.0-rootless -- build --frontend=dockerfile.v0 --local context=/home/user/files --local dockerfile=/home/user/files  --opt filename=Dockerfile --import-cache type=registry,ref=localhost:5000/buildkit-cache:cache --export-cache type=registry,ref=localhost:5000/buildkit-cache:cache,mode=max,compression=zstd,compression-level=1 --progress plain

You should see that the go download command is executed

#8 [3/4] RUN go mod download
#8 0.019 go: warning: ignoring go.mod in $GOPATH /go
#8 DONE 6.3s

Now update the last line of the dockerfile to RUN echo hello world2 and run the build again

This time the go download command will reuse the cached layer, but it will still take a lot of time to extract

#7 [2/4] COPY <<EOL go.mod
#7 CACHED

#8 [3/4] RUN go mod download
#8 sha256:369638ea322fa4eb3743bcc976731bbc20efd2a625ee66b9f23957d75aeb44e5 521B / 521B 0.0s done
#8 sha256:802482bf6331395a479ff591f73ad14f7e0c13903db55166981241fb07ee77a6 126B / 126B done
#8 sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 32B / 32B 0.0s done
#8 sha256:243a2f381e4cd3963e3af5194953e3e2807c452e833bf69397dee70610e428e6 57.66MB / 57.66MB 0.1s done
#8 sha256:c9b629762372f548de0ebccf01b8e80ae5ce251dfd36aef6fc3ae8d963493edf 67.59MB / 67.59MB 0.1s done
#8 sha256:9dbf0b8a807c2eba959326591a264b2e464d70f28bdb151bfa8892097a09d6ab 98.28MB / 98.28MB 0.1s done
#8 sha256:a80f492b9d78018f20a34c0ab51437e7c076d6788cd0bd119c76cc3043240380 103.04MB / 103.04MB 0.1s done
#8 sha256:599d5b6b6766fd729045e2e7d0396d1f61fe41c612d4aef6bb3bf5ea7db12ae2 25.02MB / 25.02MB 0.0s done
#8 sha256:5582010cab7f00a8f96e076b02666116eaa7e4af9a74eb44f2946a593b50294f 49.65MB / 49.65MB 0.0s done
#8 extracting sha256:5582010cab7f00a8f96e076b02666116eaa7e4af9a74eb44f2946a593b50294f
#8 extracting sha256:5582010cab7f00a8f96e076b02666116eaa7e4af9a74eb44f2946a593b50294f 0.4s done
#8 DONE 0.7s

#8 [3/4] RUN go mod download
#8 extracting sha256:599d5b6b6766fd729045e2e7d0396d1f61fe41c612d4aef6bb3bf5ea7db12ae2
#8 extracting sha256:599d5b6b6766fd729045e2e7d0396d1f61fe41c612d4aef6bb3bf5ea7db12ae2 0.4s done
#8 DONE 1.0s

#8 [3/4] RUN go mod download
#8 extracting sha256:c9b629762372f548de0ebccf01b8e80ae5ce251dfd36aef6fc3ae8d963493edf
#8 extracting sha256:c9b629762372f548de0ebccf01b8e80ae5ce251dfd36aef6fc3ae8d963493edf 0.7s done
#8 DONE 1.7s

#8 [3/4] RUN go mod download
#8 extracting sha256:9dbf0b8a807c2eba959326591a264b2e464d70f28bdb151bfa8892097a09d6ab
#8 extracting sha256:9dbf0b8a807c2eba959326591a264b2e464d70f28bdb151bfa8892097a09d6ab 0.9s done
#8 DONE 2.6s

#8 [3/4] RUN go mod download
#8 extracting sha256:243a2f381e4cd3963e3af5194953e3e2807c452e833bf69397dee70610e428e6
#8 extracting sha256:243a2f381e4cd3963e3af5194953e3e2807c452e833bf69397dee70610e428e6 1.0s done
#8 DONE 3.6s

#8 [3/4] RUN go mod download
#8 extracting sha256:802482bf6331395a479ff591f73ad14f7e0c13903db55166981241fb07ee77a6 done
#8 extracting sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1 done
#8 extracting sha256:369638ea322fa4eb3743bcc976731bbc20efd2a625ee66b9f23957d75aeb44e5 done
#8 extracting sha256:a80f492b9d78018f20a34c0ab51437e7c076d6788cd0bd119c76cc3043240380
#8 extracting sha256:a80f492b9d78018f20a34c0ab51437e7c076d6788cd0bd119c76cc3043240380 1.0s done
#8 DONE 4.7s

#9 [4/4] RUN echo hello world2
#9 0.015 hello world2
#9 DONE 0.0s

The above was run on my mac with a minimal Dockerfile, but on my CI server (despite high level HW) i see similar Dockerfiles take 20+ seconds to extract (which makes the cache harmful instead of useful)

Version information

Running buildkit v0.27.0-rootless