crashappsec / crashappsec/ncc

lto_guard/lto_audit: emit precise GC root maps as the long-term replacement

Open
#8 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
0
Forks
0
Avg merge
5d 4h
Merged PRs (30d)
1

Description

This is the long-term followup to the work in [#7](https://github.com/crashappsec/ncc/issues/7)
and the just-landed `--ncc-lto-guard` + `--ncc-lto-audit` flags.

## Where we are today

n00b uses a conservative copying GC. Three sources of stale pointers can sneak past it:

1. **Compiler keeps a pointer in a callee-saved register** across a possible-GC call,
so the GC's stack scan never sees the slot. → `--ncc-lto-guard` injects `volatile`
on every block-scope local pointer (and struct-field pointer), forcing the
compiler to spill across every call. Fixes class 1.

2. **A local caches a deref result that the GC later forwards in the original
container, but the cached copy isn't updated.** Volatile doesn't help — the
slot was written once with the pre-GC value. → `--ncc-lto-audit` warns on the
pattern so a human can remediate (re-deref every use, or pin).

3. **A thread is in a state where its registers can't be scanned**
(unregistered new thread, or blocked deep in a syscall the runtime can't STW).
Handled per-site (e.g., allocating thread-spawn bundles from `system_pool`).

The two flags work and are landing. But both are pattern-matched compiler band-aids
that interact awkwardly with each other and with real C semantics: `volatile`
discards qualifiers at every `&local`-passed-to-`T**` site, generates whole-build
warning noise, and inhibits perfectly safe register caching. The audit is purely
advisory and only catches the patterns we taught it to look for.

## The structural answer: precise GC root maps

This is what Go, V8, SBCL, HotSpot, OCaml's MLton and friends all do. The
compiler emits per-function metadata — a table keyed by PC range — that lists
exactly which stack offsets (and which callee-saved registers) hold GC pointers
at that program point, along with their static type IDs. At GC time, the
collector walks frames using frame pointers + dwarf, consults the per-function
map, and treats *only* those slots as roots.

What this enables:
- Drop the conservative stack scan altogether. No more pointer-shape heuristic
(no more false positives keeping garbage alive; no more false negatives losing
real pointers because they don't look pointer-shaped).
- Drop `--ncc-lto-guard`. The compiler is free to keep pointers in callee-saved
registers because the GC reads them out of the unwind state on STW.
- Drop `--ncc-lto-audit` — the stale-derived-local class disappears too, because
the GC can also update register slots and the stack slots that hold cached
derefs (those are still pointer-typed slots; the map says so).
- Precise typing means we can do moving collection on non-pointer-shaped values
(NaN-boxed unions, tagged integers) without false forwards.

## Implementation sketch

Inside ncc (where we already have all type info post-typecheck):
- Per function body, emit a `.n00b_gcmap` section (or whatever container fits
Mach-O / ELF / PE) keyed by `(function_start_pc, function_end_pc, entry_count)`.
Each entry is `(pc_range_start, pc_range_end, num_live_slots, [(fp_offset, type_id) ...])`.
- The type_id references a global type table (we have one for typeid/typestr/
typehash already; reuse the registry).
- For locals: we have their declared type and which stack slot the backend
parks them in (for that we need codegen cooperation — the host C compiler
ultimately picks the slot).
- For callee-saved registers: we need register liveness info, which is harder
in C without compiler hooks. Could compromise: still spill pointers (à la
lto_guard) but only at calls listed in the map; precise stack slot tracking
buys us most of the win without register-level precision.

In n00b's runtime:
- New STW scanner that walks frames precisely using frame pointers (n00b
already requires FP), looks up the active PC range in each frame's
function's gcmap, and reports those slots as roots to the moving collector.
- Coexist with conservative scanning during transition (use gcmap when
present, fall back conservatively otherwise).
- The forwarding pass also updates each map-listed slot.

## Why not now

This is a multi-week project that touches: parser/typecheck (to enumerate live
pointer locals at each call site), codegen (to know stack slots — this part
needs codegen cooperation if we're staying with clang as the backend), the
gcmap section format and emission, n00b's STW scanner, and the moving collector's
update pass. The current flags get us 95% of the way for the codebases that
care most, and we can ship them today.

## Acceptance / when to start

- All current lto_audit warnings in n00b/n00b-regex must be resolvable via the
precise-map mechanism (no false positives, no false negatives).
- An A/B test: build the same suite with conservative scan vs gcmap; both pass
the same tests under stress; gcmap is faster (no random-word scan) and
smaller GC pauses.

## Dependencies

- Issue #7 (lto_guard struct fields / targeted analysis) is technically
subsumed by this — once precise maps land, lto_guard is gone. Until then,
#7 keeps the band-aid honest.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.