api7 / api7/aisix

build: ship the release binary with frame pointers (-C force-frame-pointers=yes)

Open
#1,063 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
157
Forks
32
Avg merge
1h 25m
Merged PRs (30d)
145

Description

Extends the #847 profiling contract: the shipped binary keeps its symbol table
so function-level flame graphs work in the field. This asks for one more
property with the same rationale: build the release binary with
-C force-frame-pointers=yes, so the frame-pointer half of the profiling
ecosystem works on the shipped image too.

What we measured against the shipped binary

While validating an external CPU profiler against the release image (as
shipped, no rebuild), we needed an independent reference for the call chains.
Both of perf's modes failed on the shipped binary:

  • perf record --call-graph fp: unusable — the release profile does not
    emit frame pointers, so the kernel's rbp walk has nothing to follow.
  • perf record --call-graph dwarf: libdw fails to evaluate this binary's
    call-frame information. Measured on a 6.18 KVM guest under a
    bench/pgo-training load: only 22 of 618 sampled chains ever crossed
    the libc-to-aisix boundary; everything else stopped there. The CFI data
    itself is fine — readelf --debug-dump=frames-interp parses all 36,928
    FDEs
    cleanly — it is elfutils' evaluator that gives up on the fat-LTO
    output. Upstream's bug, but it ships in every distro's perf.

So today, no stock profiling tool can walk the shipped binary's stacks. To get
a working reference at all we had to rebuild the pinned release commit with
RUSTFLAGS="-C force-frame-pointers=yes". On that build, perf --call-graph fp walked 99.9% of samples to the thread entry with user-chain depth p90
of 44 — essentially perfect. One flag is the difference between "no external
tool works" and "everything fp-based works".

Cost

  • Industry measurements put force-frame-pointers at under 1% on x86-64
    (16 GPRs; losing rbp as a scratch register stopped mattering a decade
    ago). Fedora 38+, Ubuntu 24.04+ and Arch now build their entire
    distributions with frame pointers on this reasoning.
  • +8 bytes of stack per frame; binary-size delta is noise next to the
    symbol table #847 already ships (~10 MiB).
  • Fat LTO, codegen-units = 1, and PGO are unaffected — frame pointers are a
    per-function prologue property, orthogonal to inlining and layout.

What it buys

  • perf record -g (fp is its default), bcc/bpftrace stack helpers, and every
    other rbp-walking tool work on the release image as shipped — support
    and customers can profile without a rebuild.
  • Profiling validation stops rebuilding the world: the flag makes the shipped
    artifact its own reference target.

Honest limits

  • It does not fix libdw's fat-LTO problem; --call-graph dwarf stays broken
    until elfutils does.
  • rbp walkers still lose the frames below any call into a library built
    without frame pointers (today: glibc in the base image). That blind spot
    belongs to the base image's libc, not to this flag; it shrinks as distros
    ship fp-built libc.

Implementation sketch

The release binary's RUSTFLAGS are set in the Dockerfile's PGO phases
(-Cprofile-generate=... and -Cprofile-use=...) and unset in the
PGO=off branch. Append -C force-frame-pointers=yes to all three — keeping
both PGO phases' flags identical so the trained layout matches the shipped
shape — or hoist it into one ARG/ENV so it cannot drift between branches.

The #847 CI check already pins the symbol table because "a future strip step,
RUSTFLAGS change, or base-image swap would break flame graphs silently".
Frame pointers deserve the same pin once added: assert a sampled function
prologue starts with push %rbp; mov %rsp,%rbp (e.g. objdump -d on main)
in the same job, so the property cannot regress silently either.

Verification

objdump -d --no-show-raw-insn /usr/local/bin/aisix | grep -A2 '<main>:' | head -3
# expect: push %rbp / mov %rsp,%rbp

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Locate the Dockerfile's PGO generate, PGO use, and PGO=off branches, then read the existing #847 CI check for the shipped binary. Add the frame-pointer flag consistently to the release builds and extend that check with the supplied objdump verification. Done means the shipped binary's sampled function prologue shows frame-pointer setup without breaking the PGO branches.

Written by the indexing model from the issue text.

Assessment

Tech stack
dockerfile, rust
Domain
build-system, observability, performance
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.