kubernetes / kubernetes/perf-tests
Make scalability test results more accessible
- Dominant language
- Go
- Stars
- 989
- Forks
- 654
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 56
Description
**What would you like to be added**:
We (me and @Jefftree) propose making scalability test results and historical Prow run data more accessible by creating new tool within perf-test.
Currently, scalability run data is scattered across raw GCS buckets, log archives, and metrics snapshots. This enhancement introduces a lightweight, containerized tooling stack and ingestion workflow that unifies multiple data sources into a single queryable endpoint and pre-configured Grafana for dashboards.
We want to make scalability data more accessible to new contributors that could use more standardized dashboards that can be reused between investigations and agents having all the data in single place. For we are targeting local setups that you could setup via scripts or via agent skill.
Indented flow for debugging scalability tests:
1. Setup a stack via a script without any data.
2. Pick set of runs you would like to debug, and run script for each of them.
3. Open Grafana showing all the data data and dashboards.
There is nothing magical in the things above, we are just codifying the intended flow that Scalability Experts were already familiar and doing it manually. We just want to make it easier to access for everyone else.
Example dashboard I created for debugging of https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gce-scale-performance-5000/2074539413072777216 failure:
API server dahboard including stages of CL2 scenario that usually requires manually looking up logs:
Log-based metrics dashboard that usually requires grepping and logging tools. Using log traces has been the main tool for debugging high percentile latency for long time, but little known outside scalability experts.
**Why is this needed**:
Diagnosing scalability test regressions (such as failures in the GCE 5,000-node scale performance test suite) is currently fragmented, manual, and prone to diagnostic red herrings.
1. Fragmented Scale Test Diagnostics
Currently, analyzing a scale failure requires:
* Manually fetching multi-gigabyte log files and metric archives from Prow GCS buckets.
* Spinning up ad-hoc local Prometheus containers to inspect raw metrics.
* Running custom parsing logic to extract Trace logs or APF queue delays.
* Manually correlating events across different runs without side-by-side metric comparison tools.
2. Autonomous AI Diagnostics vs. Human Verification Needs
My personal experience from debugging recent scalability test failures investigations shows that AI agents attempting to diagnose scale failures in isolation suffer from diagnostic pitfalls unless supported by human verification, and humans need easily accessible verified graphs to read and check evidence.
Considerations:
* Why this repo? This tools are will be targeted for scalability and performance based debugging.
Contributor guide
Research direction
Start by inspecting the perf-test repository for existing scalability tooling and how Prow GCS logs, log archives, and metrics snapshots are accessed. Define the local containerized workflow and ingestion boundaries first; done means selected runs can be loaded into one queryable endpoint and viewed through pre-configured Grafana dashboards.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, gcp, go, grafana, prometheus
- Domain
- data-engineering, observability, performance, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100