cockroachdb / cockroachdb/cockroach

perf,o11y: expand pprof labeling by reducing cost to label

Open
#155,285 1 comment 0 reactions 0 assignees View on GitHub
C-enhancement
Dominant language
Go
Stars
32.5k
Forks
4.1k
PR merge metrics
PR metrics pending

Description

## Problem Statement

The current `runtime/pprof` label system provides a simple API (`Labels`, `WithLabels`, `Do`) for attaching metadata to goroutines for profiling.
However, the implementation incurs significant allocation overhead on request-heavy paths:

- Each `WithLabels` allocates a new context wrapper (`valueCtx`).
- Each `LabelSet` built via the variadic `Labels(kv ...string)` allocates a new slice.
- Dynamic string formatting (`strconv`) for label values adds further heap churn.

In high-QPS servers, these allocations are too costly to use `pprof` labeling per request.

We want to make label usage **cheap enough to use everywhere**, without changing any visible API or semantics.

---

## Possible Solution

We can improve the implementation incrementally while keeping the same public API behavior.

1. **Benchmark & Baseline:** Quantify current allocations for `Labels`, `WithLabels`, and `Do`.
2. **Pool `WithLabels` internals:** Reuse the internal `labelMap` and `context` wrapper via `sync.Pool`.
- `Do` automatically returns borrowed objects to the pools.
- No API or behavior change; this is a pure optimization.
3. **Mutable, Resettable `LabelSet`:**
- Add `Reset()` and `AddString()` methods.
- Users can pool their own `LabelSet`s for reuse across requests.
- Enables fully alloc-free label usage when all label strings are constants.
4. **Numeric Label Support (optional, new feature):**
- Add `AddNum()` storing an `int64` directly in each label.
- Emit stringified values (`strconv.FormatInt`) only when writing profiles.
- Makes dynamic per-request numeric labels (e.g. request IDs) alloc-free.

Each step delivers measurable improvements with low risk.

---

## Detailed Implementation Plan

### 1. Benchmark Current Allocation Cost

**Goal:** establish a baseline.

- Add benchmarks covering:
- `pprof.Labels("key", "value")`
- `pprof.WithLabels(ctx, labels)`
- `pprof.Do(ctx, labels, func(context.Context){})`
- Measure allocations/op using `testing.Benchmark`.
- Expected: several allocs per call (slice, map, context).

Deliverable: benchmark results and flamegraph showing allocation sites.

---

### 2. Pool `WithLabels` Internals (no API change)

**Goal:** make `WithLabels`/`Do` alloc-free in steady state.

- Add `sync.Pool`s for:
- `*labelMap` (holds merged `LabelSet`)
- `*labelCtx` (wrapper implementing `context.Context`)
- Modify `WithLabels`:
- Pull from pool instead of allocating new objects.
- Merge parent + child labels into pooled `labelMap`.
- Modify `Do`:
- Call `WithLabels`.
- In `defer`, reset and return both objects to pools.
- Add internal helper `ReleaseLabels(ctx)` (no-op unless it’s a pooled wrapper).
- Used by `Do` to return pooled objects.
- Optional public export if desired.

**Outcome:** identical semantics, zero allocs for pooled steady-state paths.

---

### 3. Mutable, Resettable `LabelSet` (user-side pooling)

**Goal:** let users avoid allocating new `LabelSet`s each call.

Changes:
- Add:
```go
func (ls *LabelSet) Reset()
func (ls *LabelSet) AddString(key, val string)
```
- Document that users may `sync.Pool` their own `*LabelSet` and reuse:
```go
ls := pool.Get().(*LabelSet)
ls.Reset()
ls.AddString("method", "GET")
pprof.Do(ctx, *ls, work)
pool.Put(ls)
```
- Internally, reuse existing `[]label` backing slice (no new allocs).
- Keeps same merge/sort semantics and output.

**Outcome:** fully alloc-free labeling for const strings; user code controls pooling.

---

### 4. Numeric Labeling (new feature)

**Goal:** enable alloc-free dynamic labels (e.g. `request-id`).

Additions:
- Extend internal `label`:
```go
type label struct {
key string
s string
n int64
kind uint8 // 0=string, 1=numeric
}
```
- Add:
```go
func (ls *LabelSet) AddNum(key string, val int64)
```
- Emission path:
- When `kind==1`, stringify on emit:
```go
w.pbLabel(lb.key, strconv.FormatInt(lb.n, 10), 0)
```
- Wire output remains identical (string field only).
- Request path remains alloc-free; string conversion happens only when profiles are written.

---

## Delivery Phases

| Phase | Scope | Risk | API Impact |
|-------|--------|------|------------|
| 1 | Benchmark | none | none |
| 2 | Pool `WithLabels` internals | very low | none |
| 3 | Mutable/Resettable `LabelSet` | low | additive (new methods) |
| 4 | Numeric labeling | medium (new code paths) | additive (new method) |

---

## Expected Results

- After Phase 2: `Do(ctx, Labels("k","v"), f)` allocs drop from ~5 → ~1 (only LabelSet construction).
- After Phase 3: user-pooled LabelSets reduce this to **0 allocs** for constant labels.
- After Phase 4: even dynamic numeric labels are **alloc-free** on request paths.

---

## Summary

This project incrementally modernizes `pprof` label handling to make per-request labeling practical:

1. Measure baseline allocations.
2. Pool internal objects (pure perf gain).
3. Let users pool their own `LabelSet`s (alloc-free for const labels).
4. Add numeric label support for truly zero-alloc dynamic labeling.

All existing behavior and APIs remain compatible; each step is measurable, incremental, and low risk.

Jira issue: CRDB-55343

Contributor guide

Open the contributing guide

Research direction

The named entry points are runtime/pprof's Labels, WithLabels, and Do; first inspect their current allocation behavior and add benchmarks for those calls. Compare the proposed pooling, resettable LabelSet, and optional numeric-label phases against existing API and semantics. Done means allocation costs are measured and the selected changes preserve visible behavior, with numeric values formatted only when profiles are written.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.