Design: heap profiling for snmalloc

Heap profiling: settle the design, then land it in stacked PRs

The pitch

Today, when a snmalloc process holds 8 GB, there is no built-in way to ask which lines of code allocated it.

Heap profiling answers that. Sample a small slice of allocations, keep each call stack, and emit a flame graph of live bytes. tcmalloc and jemalloc both do this. snmalloc should too.

#852 landed the first pieces (Sampler, SampledList, SampledAlloc). Review raised five design questions. They decide whether that code gets extended or rewritten, so settle them here first.

Terms

• sample
  • one recorded allocation: address, size, call stack
• fast path
  • the few instructions every malloc runs
  • cost here is paid by everyone, profiling or not
• sizeclass
  • snmalloc's bucket of same-size objects

The five questions

1. What triggers a sample?

You want to know	Right model
Where live bytes come from	Byte-weighted Poisson
Which sites allocate most often	Count-based
Bugs at every size (GWP-ASan)	Count per sizeclass

Target use case is live bytes, so byte-weighted Poisson. Same as tcmalloc and jemalloc.

2. How do we branch on the fast path?

• counter per alloc (in #852)
  • simplest
  • costs one subtract and one branch on every allocation
• freelist split
  • at slab refill, check if the batch crosses the threshold and split it
  • zero fast-path cost. More refill work
• slow path only: free, but samples land wherever refills happen

Freelist split looks best. It also means the counter in #852 goes away rather than gets built on.

3. One global sample list, or one per thread?

#852 uses a global list with a CAS on every push. Per-thread lists drop all cross-thread sync and merge only at dump time.

Per-thread fits snmalloc, which avoids global state everywhere else. One catch: thread A allocates, thread B frees. Removal must find A's list — either a back-pointer in SampledAlloc, or a message like the existing remote dealloc cache.

4. Ship extra sampling modes?

Neither tcmalloc nor jemalloc has them. We could add count-based and per-sizeclass modes. Ship byte-weighted only and revisit, or do it now?

5. Where do sampled objects live?

• main heap + side table
  • ptr → SampledAlloc* for O(1) free
  • no per-object overhead, but needs a concurrent table
• secondary allocator (like GWP-ASan, #740)
  • sampled objects go to a separate pool
  • dealloc misses the main heap and falls through
  • no side table
  • sampled frees are slower

Secondary allocator is cleaner and leaves the dealloc fast path alone.

What #852 already gets right

• Byte-weighted Poisson, matching tcmalloc and jemalloc.
• Sampler state is per-thread, so the sampling decision needs no locks.
• The interval global is read about once per 512 KB, so it is ~free.
• Stores raw addresses and resolves symbols at dump time.
• Re-entrancy guard handles backtrace() calling back into malloc.

Still to build

• Wire hooks into alloc, dealloc, and in-place realloc.
• Allocate SampledAlloc nodes without re-entering snmalloc (pre-made
  pool, mmap, or startup arena).
• O(1) freed-pointer → node lookup.
• Dump: walk live samples, symbolize, write output.
• Format: pprof, so existing pprof and flame graph tools just work.
• Rust FFI: start, stop, dump, and a raw sample stream.

Plan

A working implementation exists on restack-split as 15 reviewed commits, each building and testing on its own. Once the questions above settle, it goes up as one PR per commit, each linked here.

Related

• #852 — first infrastructure PR
• #851 — memory_stats() Rust bindings
• #740 — secondary allocator (question 5)
• #787 — heap walk (walk vs. sample)

Start by reading #852 and the 15 reviewed commits on the restack-split branch, then work through the five unresolved design questions in this issue. Done means the sampling design is settled and the remaining allocation hooks, dump format, and Rust FFI work can proceed as linked stacked PRs.