kubernetes / kubernetes/perf-tests

Make scalability test results more accessible

Open
#4,205 5 comments 0 reactions 0 assignees View on GitHub
kind/feature
Dominant language
Go
Stars
989
Forks
654
Avg merge
3d 12h
Merged PRs (30d)
56

Description

**What would you like to be added**:

We (me and @Jefftree) propose making scalability test results and historical Prow run data more accessible by creating new tool within perf-test.

Currently, scalability run data is scattered across raw GCS buckets, log archives, and metrics snapshots. This enhancement introduces a lightweight, containerized tooling stack and ingestion workflow that unifies multiple data sources into a single queryable endpoint and pre-configured Grafana for dashboards.

We want to make scalability data more accessible to new contributors that could use more standardized dashboards that can be reused between investigations and agents having all the data in single place. For we are targeting local setups that you could setup via scripts or via agent skill.

Indented flow for debugging scalability tests:
1. Setup a stack via a script without any data.
2. Pick set of runs you would like to debug, and run script for each of them.
3. Open Grafana showing all the data data and dashboards.

There is nothing magical in the things above, we are just codifying the intended flow that Scalability Experts were already familiar and doing it manually. We just want to make it easier to access for everyone else.

Example dashboard I created for debugging of https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-e2e-gce-scale-performance-5000/2074539413072777216 failure:

API server dahboard including stages of CL2 scenario that usually requires manually looking up logs:
Image

Log-based metrics dashboard that usually requires grepping and logging tools. Using log traces has been the main tool for debugging high percentile latency for long time, but little known outside scalability experts.

Image

**Why is this needed**:

Diagnosing scalability test regressions (such as failures in the GCE 5,000-node scale performance test suite) is currently fragmented, manual, and prone to diagnostic red herrings.

1. Fragmented Scale Test Diagnostics
Currently, analyzing a scale failure requires:

* Manually fetching multi-gigabyte log files and metric archives from Prow GCS buckets.
* Spinning up ad-hoc local Prometheus containers to inspect raw metrics.
* Running custom parsing logic to extract Trace logs or APF queue delays.
* Manually correlating events across different runs without side-by-side metric comparison tools.

2. Autonomous AI Diagnostics vs. Human Verification Needs

My personal experience from debugging recent scalability test failures investigations shows that AI agents attempting to diagnose scale failures in isolation suffer from diagnostic pitfalls unless supported by human verification, and humans need easily accessible verified graphs to read and check evidence.

Considerations:
* Why this repo? This tools are will be targeted for scalability and performance based debugging.

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the perf-test repository for existing scalability tooling and how Prow GCS logs, log archives, and metrics snapshots are accessed. Define the local containerized workflow and ingestion boundaries first; done means selected runs can be loaded into one queryable endpoint and viewed through pre-configured Grafana dashboards.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, gcp, go, grafana, prometheus
Domain
data-engineering, observability, performance, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.