build: ship the release binary with frame pointers (-C force-frame-pointers=yes)
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 157
- Forks
- 32
- Avg merge
- 1h 25m
- Merged PRs (30d)
- 145
Description
Extends the #847 profiling contract: the shipped binary keeps its symbol table
so function-level flame graphs work in the field. This asks for one more
property with the same rationale: build the release binary with
-C force-frame-pointers=yes, so the frame-pointer half of the profiling
ecosystem works on the shipped image too.
What we measured against the shipped binary
While validating an external CPU profiler against the release image (as
shipped, no rebuild), we needed an independent reference for the call chains.
Both of perf's modes failed on the shipped binary:
perf record --call-graph fp: unusable — the release profile does not
emit frame pointers, so the kernel's rbp walk has nothing to follow.perf record --call-graph dwarf: libdw fails to evaluate this binary's
call-frame information. Measured on a 6.18 KVM guest under a
bench/pgo-trainingload: only 22 of 618 sampled chains ever crossed
the libc-to-aisix boundary; everything else stopped there. The CFI data
itself is fine —readelf --debug-dump=frames-interpparses all 36,928
FDEs cleanly — it is elfutils' evaluator that gives up on the fat-LTO
output. Upstream's bug, but it ships in every distro'sperf.
So today, no stock profiling tool can walk the shipped binary's stacks. To get
a working reference at all we had to rebuild the pinned release commit with
RUSTFLAGS="-C force-frame-pointers=yes". On that build, perf --call-graph fp walked 99.9% of samples to the thread entry with user-chain depth p90
of 44 — essentially perfect. One flag is the difference between "no external
tool works" and "everything fp-based works".
Cost
- Industry measurements put
force-frame-pointersat under 1% on x86-64
(16 GPRs; losingrbpas a scratch register stopped mattering a decade
ago). Fedora 38+, Ubuntu 24.04+ and Arch now build their entire
distributions with frame pointers on this reasoning. - +8 bytes of stack per frame; binary-size delta is noise next to the
symbol table #847 already ships (~10 MiB). - Fat LTO,
codegen-units = 1, and PGO are unaffected — frame pointers are a
per-function prologue property, orthogonal to inlining and layout.
What it buys
perf record -g(fp is its default), bcc/bpftrace stack helpers, and every
other rbp-walking tool work on the release image as shipped — support
and customers can profile without a rebuild.- Profiling validation stops rebuilding the world: the flag makes the shipped
artifact its own reference target.
Honest limits
- It does not fix libdw's fat-LTO problem;
--call-graph dwarfstays broken
until elfutils does. - rbp walkers still lose the frames below any call into a library built
without frame pointers (today: glibc in the base image). That blind spot
belongs to the base image's libc, not to this flag; it shrinks as distros
ship fp-built libc.
Implementation sketch
The release binary's RUSTFLAGS are set in the Dockerfile's PGO phases
(-Cprofile-generate=... and -Cprofile-use=...) and unset in the
PGO=off branch. Append -C force-frame-pointers=yes to all three — keeping
both PGO phases' flags identical so the trained layout matches the shipped
shape — or hoist it into one ARG/ENV so it cannot drift between branches.
The #847 CI check already pins the symbol table because "a future strip step,
RUSTFLAGS change, or base-image swap would break flame graphs silently".
Frame pointers deserve the same pin once added: assert a sampled function
prologue starts with push %rbp; mov %rsp,%rbp (e.g. objdump -d on main)
in the same job, so the property cannot regress silently either.
Verification
objdump -d --no-show-raw-insn /usr/local/bin/aisix | grep -A2 '<main>:' | head -3
# expect: push %rbp / mov %rsp,%rbp
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate the Dockerfile's PGO generate, PGO use, and PGO=off branches, then read the existing #847 CI check for the shipped binary. Add the frame-pointer flag consistently to the release builds and extend that check with the supplied objdump verification. Done means the shipped binary's sampled function prologue shows frame-pointer setup without breaking the PGO branches.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- dockerfile, rust
- Domain
- build-system, observability, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100