cockroachdb / cockroachdb/cockroach
perf,o11y: expand pprof labeling by reducing cost to label
- Dominant language
- Go
- Stars
- 32.5k
- Forks
- 4.1k
- PR merge metrics
- PR metrics pending
Description
## Problem Statement
The current `runtime/pprof` label system provides a simple API (`Labels`, `WithLabels`, `Do`) for attaching metadata to goroutines for profiling.
However, the implementation incurs significant allocation overhead on request-heavy paths:
- Each `WithLabels` allocates a new context wrapper (`valueCtx`).
- Each `LabelSet` built via the variadic `Labels(kv ...string)` allocates a new slice.
- Dynamic string formatting (`strconv`) for label values adds further heap churn.
In high-QPS servers, these allocations are too costly to use `pprof` labeling per request.
We want to make label usage **cheap enough to use everywhere**, without changing any visible API or semantics.
---
## Possible Solution
We can improve the implementation incrementally while keeping the same public API behavior.
1. **Benchmark & Baseline:** Quantify current allocations for `Labels`, `WithLabels`, and `Do`.
2. **Pool `WithLabels` internals:** Reuse the internal `labelMap` and `context` wrapper via `sync.Pool`.
- `Do` automatically returns borrowed objects to the pools.
- No API or behavior change; this is a pure optimization.
3. **Mutable, Resettable `LabelSet`:**
- Add `Reset()` and `AddString()` methods.
- Users can pool their own `LabelSet`s for reuse across requests.
- Enables fully alloc-free label usage when all label strings are constants.
4. **Numeric Label Support (optional, new feature):**
- Add `AddNum()` storing an `int64` directly in each label.
- Emit stringified values (`strconv.FormatInt`) only when writing profiles.
- Makes dynamic per-request numeric labels (e.g. request IDs) alloc-free.
Each step delivers measurable improvements with low risk.
---
## Detailed Implementation Plan
### 1. Benchmark Current Allocation Cost
**Goal:** establish a baseline.
- Add benchmarks covering:
- `pprof.Labels("key", "value")`
- `pprof.WithLabels(ctx, labels)`
- `pprof.Do(ctx, labels, func(context.Context){})`
- Measure allocations/op using `testing.Benchmark`.
- Expected: several allocs per call (slice, map, context).
Deliverable: benchmark results and flamegraph showing allocation sites.
---
### 2. Pool `WithLabels` Internals (no API change)
**Goal:** make `WithLabels`/`Do` alloc-free in steady state.
- Add `sync.Pool`s for:
- `*labelMap` (holds merged `LabelSet`)
- `*labelCtx` (wrapper implementing `context.Context`)
- Modify `WithLabels`:
- Pull from pool instead of allocating new objects.
- Merge parent + child labels into pooled `labelMap`.
- Modify `Do`:
- Call `WithLabels`.
- In `defer`, reset and return both objects to pools.
- Add internal helper `ReleaseLabels(ctx)` (no-op unless it’s a pooled wrapper).
- Used by `Do` to return pooled objects.
- Optional public export if desired.
**Outcome:** identical semantics, zero allocs for pooled steady-state paths.
---
### 3. Mutable, Resettable `LabelSet` (user-side pooling)
**Goal:** let users avoid allocating new `LabelSet`s each call.
Changes:
- Add:
```go
func (ls *LabelSet) Reset()
func (ls *LabelSet) AddString(key, val string)
```
- Document that users may `sync.Pool` their own `*LabelSet` and reuse:
```go
ls := pool.Get().(*LabelSet)
ls.Reset()
ls.AddString("method", "GET")
pprof.Do(ctx, *ls, work)
pool.Put(ls)
```
- Internally, reuse existing `[]label` backing slice (no new allocs).
- Keeps same merge/sort semantics and output.
**Outcome:** fully alloc-free labeling for const strings; user code controls pooling.
---
### 4. Numeric Labeling (new feature)
**Goal:** enable alloc-free dynamic labels (e.g. `request-id`).
Additions:
- Extend internal `label`:
```go
type label struct {
key string
s string
n int64
kind uint8 // 0=string, 1=numeric
}
```
- Add:
```go
func (ls *LabelSet) AddNum(key string, val int64)
```
- Emission path:
- When `kind==1`, stringify on emit:
```go
w.pbLabel(lb.key, strconv.FormatInt(lb.n, 10), 0)
```
- Wire output remains identical (string field only).
- Request path remains alloc-free; string conversion happens only when profiles are written.
---
## Delivery Phases
| Phase | Scope | Risk | API Impact |
|-------|--------|------|------------|
| 1 | Benchmark | none | none |
| 2 | Pool `WithLabels` internals | very low | none |
| 3 | Mutable/Resettable `LabelSet` | low | additive (new methods) |
| 4 | Numeric labeling | medium (new code paths) | additive (new method) |
---
## Expected Results
- After Phase 2: `Do(ctx, Labels("k","v"), f)` allocs drop from ~5 → ~1 (only LabelSet construction).
- After Phase 3: user-pooled LabelSets reduce this to **0 allocs** for constant labels.
- After Phase 4: even dynamic numeric labels are **alloc-free** on request paths.
---
## Summary
This project incrementally modernizes `pprof` label handling to make per-request labeling practical:
1. Measure baseline allocations.
2. Pool internal objects (pure perf gain).
3. Let users pool their own `LabelSet`s (alloc-free for const labels).
4. Add numeric label support for truly zero-alloc dynamic labeling.
All existing behavior and APIs remain compatible; each step is measurable, incremental, and low risk.
Jira issue: CRDB-55343
Contributor guide
Research direction
The named entry points are runtime/pprof's Labels, WithLabels, and Do; first inspect their current allocation behavior and add benchmarks for those calls. Compare the proposed pooling, resettable LabelSet, and optional numeric-label phases against existing API and semantics. Done means allocation costs are measured and the selected changes preserve visible behavior, with numeric values formatted only when profiles are written.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100