metrics for cache size / prune pauses

Why

We've seen issues where buildkit will sometimes pause for a long time while the cache manager is doing a prune().

It's fairly straightforward to reproduce this in isolation. A test that creates 1,000 cache entries then garbage collects them will hold the cache lock for about ~15s on my machine. But we don't have good visibility into how frequent these pauses are in a live system.

Feature request

Can we add a prometheus handler that exposes some metrics about cache / prune behavior?

A good place to start would be to take some inspiration from the pause metrics that Go itself uses for its GC (PauseTotalNs and PauseNs in https://pkg.go.dev/runtime#MemStats)

• A Gauge metric that tracks how many cache records are currently tracked by the CacheManager
• A Counter metric that tracks how much time, cumulatively, have been spent prune()
• A Histogram metric that tracks how long a single prune() takes (maybe 0.1s, 1s, 10s, 100s+)