Caches page loading terribly slow on large sites

Adding a comment here to avoid issue bloat as it might be related to the changes here- @leonstafford in your latest 7.1.7 dev, I've noticed the Caches page is taking a reaaaaaally long time to load in some instances. After deploying a site with ~6500 URLs with a ~500mb cache for the crawled and post-processed site, the cache page takes a solid 3-4 minutes to load. After clearing the cache it loads fairly promptly. Maybe a super-expensive aggregation is occurring somewhere that gets worse the larger the cache is? Not sure - beefed up the container I'm running in to check but seems not to make a difference.

_Originally posted by @petewilcock in https://github.com/leonstafford/wp2static/issues/745#issuecomment-789576817_