kubernetes / kubernetes/perf-tests
benchmark-list: an allocation improvement shows up as a memory regression
- Dominant language
- Go
- Stars
- 989
- Forks
- 654
- Avg merge
- 3d 12h
- Merged PRs (30d)
- 56
Description
The recent ~100 MiB step in benchmark-list apiserver memory ([perfdash](https://perf-dash.k8s.io/#/?jobname=benchmark%20list&metriccategoryname=E2E&metricname=Resources&Resource=memory&PodName=kube-apiserver-benchmark-list-master%2Fkube-apiserver)) turned out to be benign GC headroom, but proving that took a manual dig into Prometheus snapshots because the dashboard only carries one number. Filing this to improve what we monitor so the next step is self-diagnosing.
**What happened**
Memory stepped 2685 to 2785 MiB and stayed there. It bisects cleanly to https://github.com/kubernetes/kubernetes/pull/139617 (JSON list encoder), which reuses one buffer instead of allocating per item, about 600x less garbage per list. The RSS went up because of that: less allocation means GC fires ~6x less often, so under GOGC=100 the heap sits higher between collections.
Runs: [good](https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-benchmark-list/2065489118564978688) (alpha.1.110) to [bad](https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-benchmark-list/2065549768825769984) (alpha.1.127).
The live heap never moved, which is the whole point and is invisible on the current dashboard:
| 8-min avg, good vs bad | good | bad |
|---|---|---|
| allocation rate | 612 MB/s | 105 MB/s |
| GC frequency | 28.5/min | 4.9/min |
| **heap_inuse (live)** | **1785 MB** | **1778 MB** |
| working_set (what we chart) | up ~100 MiB | |
**The gap**
For this job perfdash carries a single apiserver memory series, `container_memory_working_set_bytes` from `ResourceUsageSummary`, which is essentially RSS. working_set moves with GC pacing, not just real memory use, so on its own it cannot distinguish:
- a real leak or live-heap growth (the thing we want to catch), from
- GC headroom shifting because allocation rate changed (this episode, benign).
It fails the other way too: a real leak can hide under a flat working_set if GC compensates. So every allocation-pattern change risks a false alarm, and real regressions can be masked.
**Ideas**
We can surface additional metrics that are already scraped:
- live heap (`go_memstats_heap_inuse_bytes`), flat here
- GC rate (`rate(go_gc_duration_seconds_count[5m])`), dropped 6x
- allocation rate (`rate(go_memstats_alloc_bytes_total[5m])`), dropped 6x
Sidenote: the proto variant dashboard also shows a "regression" from an allocation improvement (https://github.com/kubernetes/kubernetes/pull/139350), so this is a recurring pattern of allocation improvements reading as memory regressions on the dashboard.
Contributor guide
Assessment
This issue has not been assessed yet.