Proposal: Rewire CL2 to drop MetricsForE2E in favor of Prometheus

## Summary

`MetricsForE2E` is one of CL2's older measurements. At the end of each phase it scrapes `/metrics` off the apiserver once and dumps the raw response to JSON. It was added back when CL2 had no Prometheus stack, but that hasn't been the case for a while. CL2 now scrapes those same endpoints into Prometheus every 5 seconds, so the JSON is really just a lower-resolution subset of the snapshot, and the only thing still reading it is perfdash, for a single metric.

It is also a maintenance burden. It's custom Go that assumes a specific control-plane layout, so we have to update it whenever that changes and is hard to test. Introducing HA falls into that category right now. I'm proposing we move perfdash's last metric to Prometheus so we can avoid maintaining the legacy scraper.

Steps:

1. Update perfdash to read from Prometheus, repointing the `InitEventsCount` panels at a `GenericPrometheusQuery` for `apiserver_init_events_total`.
2. Turn on the Prometheus and snapshot flags on the jobs that do not already produce a snapshot.
3. Can we then drop `MetricsForE2E` from the `TestMetrics` bundle so we no longer have to maintain the legacy scraper?

## Metrics

The grabber reaches for five buckets but most are currently empty. Numbers are from the passing 5k run [#2078163233406980096](https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gce-scale-performance-5000/2078163233406980096), file `MetricsForE2E_load_*.json`:

| Bucket | Keys |
|---|---|
| `APIServerMetrics` | 406 |
| `SchedulerMetrics` | 0 |
| `ControllerManagerMetrics` | 0 |
| `ClusterAutoscalerMetrics` | 0 |
| `KubeletMetrics` | 0 |

The scheduler, KCM and CA buckets only fill when there is a node whose name ends in `master` ([`deprecatedMightBeMasterNode`](https://github.com/kubernetes/perf-tests/blob/80909fa7adee171da0c0750d63062b9d4bcb47c9/clusterloader2/pkg/measurement/common/metrics/metrics_grabber.go#L60-L72)), which never happens on kops, so they come back empty. This looks like a leftover from the old kube-up setup. In practice only the apiserver metrics get collected.

Those 406 apiserver families expand to 37,140 series:

| Group | Series | Families |
|---|---|---|
| `apiserver_*` | 23,917 | 92 |
| `etcd_*` (apiserver client) | 9,099 | 9 |
| `apiserver_flowcontrol_*` (APF) | 1,476 | 49 |
| `workqueue_*` | 898 | 11 |
| `rest_client_*`, auth, aggregator, apiextensions, other | 1,281 | 80 |
| `go_*` / `process_*` | 233 | 165 |

## Perfdash

perfdash reads only a small subset of these metrics. It has 8 panels backed by `MetricsForE2E`, in two groups.

The 4 `RequestCountByClient` panels look up JSON key `apiserver_request_count` ([parser.go L211](https://github.com/kubernetes/perf-tests/blob/80909fa7adee171da0c0750d63062b9d4bcb47c9/perfdash/parser.go#L199-L213)). That metric was renamed to `apiserver_request_total` years ago, so these panels have been empty ever since and can just be deleted.

The 4 `InitEventsCount` panels read `apiserver_init_events_total` ([parser.go L260](https://github.com/kubernetes/perf-tests/blob/80909fa7adee171da0c0750d63062b9d4bcb47c9/perfdash/parser.go#L248-L262)):

```
sum by (resource) (apiserver_init_events_total)
```

This can be wrapped in a `GenericPrometheusQuery` and the four panels repointed ([config.go](https://github.com/kubernetes/perf-tests/blob/80909fa7adee171da0c0750d63062b9d4bcb47c9/perfdash/config.go#L260-L268)). After that nothing reads the scraper.

## Existing Jobs Compatibility

perfdash reads JSON artifacts, not Prometheus, so every relevant job has to actually write the snapshot. It is written only under `--experimental-prometheus-snapshot-to-report-dir`. The jobs that run `load` without a snapshot today fall into two groups:

| Jobs | State | Fix |
|---|---|---|
| `pull-perf-tests-clusterloader2*`, `ci-kubernetes-kubemark-{100,500}-gce`, `ci-golang-tip-k8s-master` | Prometheus already runs, snapshot is thrown away at teardown | add `--experimental-prometheus-snapshot-to-report-dir`, one line each |
| `ci-kubernetes-e2e-azure-scalability`, `pull-cluster-api-provider-azure-load-test-custom-builds` | no Prometheus at all | enable Prometheus, then add `--experimental-prometheus-snapshot-to-report-dir` |

Start with perfdash/parser.go and perfdash/config.go to trace the existing MetricsForE2E panels and the proposed Prometheus query. Review metrics_grabber.go and the TestMetrics bundle before checking the listed jobs' load and snapshot settings. Done means the obsolete request panels and scraper are removed, InitEventsCount uses Prometheus, and every relevant job writes a snapshot.

Written by the indexing model from the issue text.