Buildkit healthcheck fatally fails after cache reaches maximum keepStorage size

Start by reproducing the healthcheck failure with the reported BuildKit v0.11.6 Kubernetes driver and the supplied garbage-collection configuration. Investigate the healthcheck and garbage-collector behavior as the cache reaches keepStorage; done means identifying the failure cause and confirming that the healthcheck remains healthy at the storage limit.

Hello,

we are having issues with the buildkit builders when the cache reaches the maximum storage size. We have enabled buildkit garbage collector with the default configuration but it didn't work. We tried doubling the cache storage size using 20% of disk and also didn't work. Now our configuration is configuring the keepBytes to 800Gb which is 80% of our disk.

When the cache sizes reaches the maximum we start to receive healthcheck failed logs, here are some samples:

time="2023-05-11T18:07:30Z" level=error msg="healthcheck failed fatally" spanID=dbd0aba48dac03fb traceID=88cae16699d7657c164ef8bf1a3f1e4f`
---
time="2023-05-12T15:11:48Z" level=warning msg="healthcheck failed" actualDuration="300.749µs" spanID=ffad4b24e0973b72 timeout=30s traceID=58061902f1c41dccf66948448e6bbc10

Driver: kubernetes priviledge mode

ID:		spp6db5io375cx1k1c6u6ei1m
Platforms:	linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/arm64,linux/riscv64,linux/ppc64le,linux/s390x,linux/386,linux/mips64le,linux/mips64,linux/arm/v7,linux/arm/v6
BuildKit:	github.com/moby/buildkit v0.11.6 2951a28cd7085eb18979b1f710678623d94ed578
Labels:
	org.mobyproject.buildkit.worker.executor:		oci
	org.mobyproject.buildkit.worker.hostname:		buildkit-builders0-5b487956d9-mvdk4
	org.mobyproject.buildkit.worker.network:		host
	org.mobyproject.buildkit.worker.oci.process-mode:	sandbox
	org.mobyproject.buildkit.worker.selinux.enabled:	false
	org.mobyproject.buildkit.worker.snapshotter:		overlayfs
GC Policy rule#0:
	All:		false
	Filters:	type==source.local type==exec.cachemount type==source.git.checkout
	Keep Duration:	48h0m0s
	Keep Bytes:	512MB
GC Policy rule#1:
	All:		true
	Keep Duration:	24h0m0s
GC Policy rule#2:
	All:		true
	Keep Bytes:	800GB

toml configuration

debug = true
gckeepstorage = 800000000000
[worker.oci]
  enabled = true
  gc = true
  max-parallelism = 16 # bump from 4 as we have bigger boxes
[[worker.oci.gcpolicy]]
  keepBytes = 512000000 # default 512M
  keepDuration = 172800 # default 48h
  filters = [ "type==source.local", "type==exec.cachemount", "type==source.git.checkout"]
[[worker.oci.gcpolicy]]
  all = true
  keepDuration = 86400 # 1 day in seconds
[[worker.oci.gcpolicy]]
  all = true
  keepBytes = 800000000000 # 800GB

Could you guys help find what is wrong?