bytedance / bytedance/sonic

arm64: CPU profiling signal in asm2asm natives can abort the process with "traceback stuck"

Open
#970 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
9.6k
Forks
473
Avg merge
2d 16h
Merged PRs (30d)
2

Description

## Summary

On arm64, a CPU profiling signal landing inside sonic's `asm2asm`-generated natives can abort the whole process with `fatal error: traceback stuck`. This is a Go runtime bug rather than a sonic bug, and the runtime fix is in review — but sonic is what appears in the crash dump, so users hitting it will land here first.

Filing this mainly so it is searchable, and to record a fix that looks obvious but **does not work**, so nobody else spends a day on it.

No action is required from sonic. Details below in case you want them.

## What happens

We run continuous CPU profiling across an arm64 fleet. A process died with:

```
runtime: traceback stuck. pc=0x100000 sp=0x4b7900f214b0
stack: frame={sp:0x4b7900f214b0, fp:0x4b7900f214b0} stack=[0x4b7900f1e000,0x4b7900f22000)
...
^
^
^
fatal error: traceback stuck

goroutine 0 [idle]:
runtime.throw(...)
runtime.(*unwinder).next(...) runtime/traceback.go:482
runtime.tracebackPCs(...) runtime/traceback.go:624
runtime.sigprof(...) runtime/proc.go:5821
runtime.sighandler(...)
runtime.sigtrampgo(...)
runtime.sigtramp()
```

The Go-visible entry point is frameless and reaches the generated blob by a jump rather than a call:

```asm
TEXT ·__parse_with_padding(SB), NOSPLIT, $0-16
MOVD ·_subr__parse_with_padding(SB), R11
WORD $0x1000005e // adr x30, .+8
JMP (R11)
```

The blob then runs a C prologue the Go assembler never saw, since it is emitted as raw `WORD`s:

```asm
sub sp, sp, #32
stp fp, lr, [sp, #8]
sub fp, sp, #8 // deliberately misaligned
```

Partway through that prologue the unwinder sees `sp == fp` with an instruction word where the return address belongs. It cannot advance, and `(*unwinder).next` throws — even though `sigprof` explicitly initializes the unwinder with `unwindSilentErrors` so that a profiling signal can never kill the process. That missing flag check is the actual defect.

- Go issue: https://github.com/golang/go/issues/80771
- Go fix under review: https://go-review.googlesource.com/c/go/+/811580

Anyone hitting this today can work around it by disabling CPU profiling, or by carrying that CL.

## The fix that does not work: marking the entry symbols SPWRITE

Go has a mechanism that looks purpose-built for this. `FuncFlagSPWrite` means "this function moves SP in a way the unwinder cannot model, stop here", and arm64's assembler infers it automatically from any instruction writing `RSP` (`cmd/internal/obj/arm64/obj7.go`, "auto-SPWRITE"). Because the blob's SP writes are raw `WORD`s, the assembler never sees them, so the flag is never set on a symbol that plainly qualifies.

Making it visible is a three-line change to `tools/asm2arm/arm.py`, appending an unreachable marker after the instruction dump in `_declare_body` so subroutine offsets do not shift:

```asm
MOVD RSP, R16
MOVD R16, RSP
```

That does set the flag on all 23 `*_entry__` symbols, and it does fix the profiling crash on stock, unpatched Go — I verified both with a standalone reproducer.

**But it breaks GC.** SPWRITE tells the unwinder to stop, which is fine for a profiling sample but not for a precise stack scan, which has to traverse the whole goroutine stack. Go treats a non-innermost SPWRITE frame as fatal, and these symbols *are* non-innermost, because the blob makes internal `BL` calls — so the same symbol appears both as the innermost frame and as its own caller.

Running the suite with the marker applied, against `go1.26.5 darwin/arm64`:

```
traceback: unexpected SPWRITE function github.com/bytedance/sonic/internal/native/neon.__quote_entry__
fatal error: traceback
FAIL github.com/bytedance/sonic/internal/encoder 1.478s

traceback: unexpected SPWRITE function github.com/bytedance/sonic/internal/native/neon.__f32toa_entry__
fatal error: traceback
FAIL github.com/bytedance/sonic/internal/native/neon 1.040s
```

Both pass on the same commit without the marker. `TestMain` in those packages runs `runtime.GC()` in a loop, which is what makes it deterministic rather than rare. So the marker trades a rare profiling crash for a reliable GC crash under load — strictly worse.

## If you ever do want to fix this in sonic

The durable fix is to give the blob real unwind information rather than to tell the unwinder to give up: PC → SP delta covering the C prologue, so both the profiler and the GC can walk through it.

Worth noting that the amd64 path does not appear to have this hazard, and the reason looks structural. amd64 embeds the natives as data (`*_text_amd64.go`) and installs them through `loader`, whose `Func` carries a `Pcsp *Pcdata // PC -> SP delta` field for exactly this metadata. arm64 emits them as in-binary `.s` and relies on the Go assembler to derive spdelta, which it cannot do from `WORD` encodings. If the arm64 generator could emit the prologue's SP adjustments as real instructions the assembler understands, or otherwise supply the deltas, the frames would become properly unwindable.

That is a much larger change than this crash justifies on its own, especially since the runtime fix removes the crash regardless. Recording it as the real answer rather than a recommendation.

## Environment

- sonic v1.15.2 (also inspected `main` @ 2a36d6d)
- linux/arm64 in production, reproduced and tested on `go1.26.5 darwin/arm64`
- Go 1.26.x

Co-authored-by: Claude Opus 5

Contributor guide

Open the contributing guide

Research direction

No sonic change is requested; start with the linked Go issue and review the generated arm64 path in tools/asm2arm/arm.py, including the entry symbols and raw WORD prologue. If pursuing a sonic fix, compare the amd64 loader metadata with the arm64 output and use the native/neon and encoder test suites; done requires unwind information that works for both profiling and GC.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.