[Analysis] WatchCacheStallResume at 5k OSS scalability test

Instead of closing a watcher whose input channel fills, park it and resume it from the watch
cache history. Analysis of https://github.com/kubernetes/kubernetes/pull/141228 on 5k test

tldr:

- Endpointslice watcher terminations: **6,270 to 33,079 per run, to zero.** (significant)
- Endpointslice relist volume: **0.83M to 2.77M init-events, to 0.048M.** (significant)
- kube-proxy dataplane resyncs: **36.6 to 32.9 per proxy per minute, to 30.2.** (slight improvement)
- CPU, RSS, heap and etcd RSS have no significant changes.
- Untested: the expiry path never executed.

It seems like a pretty straight win for the shape of our 5k test. Would be interesting to see how it performs in a degraded cluster state since we're not stressing the cluster that much at the moment.

## Setup

| | build | date | code |
|---|---|---|---|
| periodic | [2085048580145418240](https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gce-scale-performance-5000/2085048580145418240) | 08-05 | master, gate off |
| periodic | [2085773363078238208](https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gce-scale-performance-5000/2085773363078238208) | 08-07 | master, gate off |
| periodic | [2086498141942583296](https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gce-scale-performance-5000/2086498141942583296) | 08-09 | master, gate off |
| **gate on** | [2087591049298972672](https://prow.k8s.io/view/gs/kubernetes-ci-logs/pr-logs/pull/141322/pull-kubernetes-gce-master-scale-performance-5000/2087591049298972672) | 08-12 | PR 141322, gate on |

`ci-kubernetes-e2e-gce-scale-performance-5000` for the periodics,
`pull-kubernetes-gce-master-scale-performance-5000` for the gate-on run. This is a sample size of 1 but the results look pretty convincing.

## Per resource

| | gate-off, 3 runs | gate-on |
|---|---|---|
| **endpointslices** | | |
| watchers killed | 6,270 - 33,079 | **0** |
| stalls resumed | n/a | **1,059** |
| init-events (relist volume) | 0.83 - 2.77M | **0.048M** |
| **pods** | | |
| watchers killed | 111 - 233 | **0** |
| stalls resumed | n/a | **4,571** |
| init-events (relist volume) | 40.7 - 83.6M | **39.6M** |

## Over time (per min)

apiserver_terminated_watchers_total. Gate-off periodics kill 6,270 to 33,079 per run. The gate-on line is flat at zero.

apiserver_watch_cache_watcher_stalls_total. Gate-on run only.

## Relists and dataplane resyncs

A sync is kube-proxy reprogramming the node's nftables rules, which it does when its
endpointslice data changes. Fewer kills -> fewer relists ->
fewer syncs, measured per proxy per minute across 1000 of the 5000 proxies.

| run | eps kills | eps init-events | peak init-events/min | syncs/min per proxy, p5 / median / p95 |
|---|---|---|---|---|
| 08-05 | 25,517 | 2,772,779 | 529,000 | 35.8 / **36.59** / 37.2 |
| 08-07 | 33,079 | 2,770,579 | 427,000 | 35.0 / **35.85** / 36.4 |
| 08-09 | 6,270 | 826,058 | 586,000 | 32.2 / **32.86** / 33.4 |
| **gate on** | **0** | **47,524** | **2,231** | 29.4 / **30.15** / 30.7 |

## Cost

Extra work from catch up:

| | gate-on |
|---|---|
| catch-up rounds served from history | 5,637 |
| events deferred to a catch-up | 146,717 |
| events streamed during catch-up | 170,058 |

Resource use:

| | gate-off, 3 runs | gate-on |
|---|---|---|
| apiserver CPU p99, cores | 24.2 - 28.9 | 29.4 |
| apiserver RSS p99, GB | 60.4 - 63.4 | 63.7 |
| apiserver heap in-use p99, GB | 49.3 - 51.6 | 53.4 |
| goroutines p99, k | 394 - 397 | 396 |
| live WATCH p99, k | 181 - 182 | 182 |
| etcd RSS p99, GB | 5.92 - 7.21 | 7.24 |
| etcd DB p99, GB | 5.54 - 6.80 | 6.80 |

Don't see anything significant changes.

## What is not established

- **The expiry path never executed.** A watcher that falls out of the history window is
supposed to get a 410 and re-list. That never happened here, so the path is untested at
scale.

The issue names no source file or test entry point. Start with PR 141228 and the 5k scale-test logs listed under Setup, then determine how to exercise the untested watch-cache expiry path and degraded-cluster behavior. Done means reporting results for those scenarios.

Written by the indexing model from the issue text.