Prometheus snapshot not created when kubetest times out

Prometheus snapshot is not created if kubetest times out. Snapshotting logic lives inside clusterloader, so when kubetest times out the logic is simply not executed. It is unfortunate, especially timeouts are situation, which we'd like to debug usually.

I was hit by this issue when tryring to debug: https://prow.k8s.io/view/gcs/kubernetes-jenkins/pr-logs/pull/88342/pull-kubernetes-e2e-gce-large-performance/1230477531426066432/

I see two options: 1) move snapshotting outside of the test (e.g. similarly to log dumping) 2) reconsider using Cortex (or any other solution that allows live recording of metrics).

@mm4tt - WDYT?

/priority important-soon
/area clusterloader

Start by examining the snapshotting logic in clusterloader and compare its execution path with log dumping, especially when kubetest times out. Decide whether snapshotting should move outside the test or use a live-recording approach; done means a Prometheus snapshot is available after a timeout.

Written by the indexing model from the issue text.