kubernetes / kubernetes/perf-tests

benchmark-list: an allocation improvement shows up as a memory regression

Open
#4,140 2 comments 0 reactions 1 assignee Claimed by @Gaurav598 View on GitHub
Dominant language
Go
Stars
989
Forks
654
Avg merge
3d 12h
Merged PRs (30d)
56

Description

The recent ~100 MiB step in benchmark-list apiserver memory ([perfdash](https://perf-dash.k8s.io/#/?jobname=benchmark%20list&metriccategoryname=E2E&metricname=Resources&Resource=memory&PodName=kube-apiserver-benchmark-list-master%2Fkube-apiserver)) turned out to be benign GC headroom, but proving that took a manual dig into Prometheus snapshots because the dashboard only carries one number. Filing this to improve what we monitor so the next step is self-diagnosing.

**What happened**

Memory stepped 2685 to 2785 MiB and stayed there. It bisects cleanly to https://github.com/kubernetes/kubernetes/pull/139617 (JSON list encoder), which reuses one buffer instead of allocating per item, about 600x less garbage per list. The RSS went up because of that: less allocation means GC fires ~6x less often, so under GOGC=100 the heap sits higher between collections.

Runs: [good](https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-benchmark-list/2065489118564978688) (alpha.1.110) to [bad](https://prow.k8s.io/view/gs/kubernetes-ci-logs/logs/ci-kubernetes-benchmark-list/2065549768825769984) (alpha.1.127).

Image

The live heap never moved, which is the whole point and is invisible on the current dashboard:

| 8-min avg, good vs bad | good | bad |
|---|---|---|
| allocation rate | 612 MB/s | 105 MB/s |
| GC frequency | 28.5/min | 4.9/min |
| **heap_inuse (live)** | **1785 MB** | **1778 MB** |
| working_set (what we chart) | up ~100 MiB | |

**The gap**

For this job perfdash carries a single apiserver memory series, `container_memory_working_set_bytes` from `ResourceUsageSummary`, which is essentially RSS. working_set moves with GC pacing, not just real memory use, so on its own it cannot distinguish:

- a real leak or live-heap growth (the thing we want to catch), from
- GC headroom shifting because allocation rate changed (this episode, benign).

It fails the other way too: a real leak can hide under a flat working_set if GC compensates. So every allocation-pattern change risks a false alarm, and real regressions can be masked.

**Ideas**

We can surface additional metrics that are already scraped:

- live heap (`go_memstats_heap_inuse_bytes`), flat here
- GC rate (`rate(go_gc_duration_seconds_count[5m])`), dropped 6x
- allocation rate (`rate(go_memstats_alloc_bytes_total[5m])`), dropped 6x

Sidenote: the proto variant dashboard also shows a "regression" from an allocation improvement (https://github.com/kubernetes/kubernetes/pull/139350), so this is a recurring pattern of allocation improvements reading as memory regressions on the dashboard.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.