How can I debug slow caches?

Hi!

I noticed our github cache action has begun to take a considerably much longer time; what tools are there available to try and investigate this?

For instance here are two post actions for saving the cache:

https://github.com/openxla/stablehlo/actions/runs/7784115717/job/21224022761
Cache size 954MiB takes 1min32s to upload in GitHub Action

https://github.com/openxla/stablehlo/actions/runs/7997664838/job/21842544903
Cache size 1852MB takes 57m to upload GitHub action

There isn't much information to understand why the increase of cache size (2x) corresponds to such a dramatic increase in upload time.

Additionally, restoring the cache seems to have doubled;

https://github.com/openxla/stablehlo/actions/runs/7924329858/job/21635691286
(~20mins)

Received 1943706716 of 1943706716 (100.0%), 231.6 MBs/sec

Fri, 16 Feb 2024 00:51:50 GMT
/usr/bin/tar: ../../../.cache/bazel/_bazel_runner/install/14fb027596f626f2526df4873ea20b8b/platforms/os/BUILD: time stamp 2034-02-12 22:39:36 is 315352065.537809154 s in the future
Fri, 16 Feb 2024 01:09:04 GMT
Cache restored successfully
Fri, 16 Feb 2024 01:09:04 GMT
Cache restored from key: Linux-stablehlo_build_assets-d592c8ec8f7138dcbde6f0890d048e59cba95041-bazelbuild

1.8 GiB takes nearly 20 mins

https://github.com/openxla/stablehlo/actions/runs/7715868908/job/21031359892

Tue, 30 Jan 2024 18:38:03 GMT
/usr/bin/tar: ../../../.cache/bazel/_bazel_runner/install/f0478d8547df16461adab7eb1d945d57/xcode-locator: time stamp 2034-01-26 12:04:44 is 315250001.01019758 s in the future
Tue, 30 Jan 2024 18:38:03 GMT
Received 995097659 of 995097659 (100.0%), 315.7 MBs/sec
Tue, 30 Jan 2024 18:38:36 GMT
Cache restored successfully
Tue, 30 Jan 2024 18:38:37 GMT
Cache restored from key: Linux-stablehlo_build_assets-98509c7f9792c79b05a41b95c24607f6dd489c5a-bazelbuild

~1GiB takes 30 seconds?

Start by comparing the upload and restore timings in the linked GitHub Actions runs and reviewing the cache log excerpts, including the tar timestamp warnings. Done means identifying a supported way to investigate the variable cache performance and documenting the relevant findings or next diagnostic steps.