FEX-Emu / FEX-Emu/FEX

Thunks: stack-protector prologue clobbers r11 (the host callee address) in CallHostFunction for >=7-argument functions

Open
#5,878 0 comments 0 reactions 0 assignees View on GitHub
ml-report
Dominant language
C++
Stars
8k
Forks
351
Avg merge
12h 31m
Merged PRs (30d)
102

Description

## Summary

`CallHostFunction` in `ThunkLibs/include/common/Guest.h` passes the **host callee address to the host side in guest `r11`** via a custom ABI. For thunked functions with **7 or more arguments**, every other register is already live, so the compiler's stack-protector prologue picks `r11` as its scratch register and emits

```asm
mov r11, fs:[0x28] ; load stack canary -> DESTROYS the host callee address
```

**before** the `asm volatile("" : "=r"(host_addr))` barrier that is supposed to pin it. The canary is then stored into the `host_addr` slot, the host side does `blr x8` on it, and the process dies with an access violation at a garbage address.

FEX's own source comment (around `Guest.h:117`) already predicts this hazard for functions with many arguments; the barrier as written is not sufficient.

## Why it is easy to miss

The faulting address is the **glibc stack canary**, so it:

- is high-entropy and **different on every run** (looks like random corruption, not a deterministic bug),
- **always ends in a zero byte** (glibc canaries are `0x??????????????00`),
- surfaces to the emulated app as a crash "in the graphics driver", pointing at the wrong component.

In our case it presented as a crash inside Mesa's venus Vulkan driver at `vn_GetQueryPoolResults`. A gdb breakpoint on that function, armed 2.5 s before the fault, **never fired** - the driver was never entered. The reported `ip` was the intended callee, not the actual PC.

## Evidence

`libGL-guest.so`, affected instantiation, before the fix:

```asm
movq %fs:0x28, %r11 ; canary load destroys the incoming callee address
movq %r11, 0x38(%rsp) ; canary slot
movq %r11, 0x30(%rsp) ; host_addr field <== host side does blr on this
```

Host side (`Host.h`) then does `ldur x8, [x0, #]` / `blr x8` and jumps into the canary.

Caught with `catch signal SIGSEGV` conditioned on `si_addr == __stack_chk_guard`:

```
__stack_chk_guard = 0xaf84028d590ad200
pc 0xaf84028d590ad200 <- PC == si_addr => instruction fetch at the canary
x8 0xaf84028d590ad200 <- branch-target register
sp 0x80089e6cfe10 <- sane; all eight arguments correct
```

Backtrace frame #1 is `GuestWrapperForHostFunction<...>::Call<...>` in the thunk library - the thunk itself, not the driver.

## Scope in our build

Instantiations of `CallHostFunction` whose prologue loads the canary into `r11` before the callee use:

| guest thunk lib | instantiations | affected |
|---|---|---|
| `libGL-guest.so` | 736 | **66** |
| `libvulkan-guest.so` | 476 | **9** (incl. `vkGetQueryPoolResults`, `vkCmdPipelineBarrier`, `vkCmdBlitImage`, `vkCmdWaitEvents`, `vkCmdCopyQueryPoolResults`) |
| `libcuda-guest.so` | 363 | **5** |
| EGL / drm / wayland-client / asound / VDSO | - | 0 |

All affected take >= 7 arguments, consistent with the register-pressure explanation.

## Fix

`__attribute__((no_stack_protector))` on `CallHostFunction` resolves it. After rebuilding, **0** instantiations load the canary into `r11`, the incoming `r11` flows straight to the `host_addr` slot, and exported symbol sets are byte-identical (verified by diffing symbol *names*, not just counts).

A more targeted alternative, if stripping the mitigation is undesirable, is to force the value into a register the prologue cannot reuse:

```cpp
register uintptr_t p asm("r11");
asm volatile("" : "=r"(p));
```

Happy to open a PR with whichever approach maintainers prefer.

## Host side

We audited the mirror-image ABI on the aarch64 host side (`Host.h`, `LOAD_INTERNAL_GUESTPTR_VIA_CUSTOM_ABI` reading `x11`). All 42 `CallGuestPtr` instantiations read `x11` before any write, so it does **not** reproduce there - but for incidental reasons: the aarch64 canary sequence allocates `x8`/`x9`, and AAPCS64 passes 8 args in registers so the starvation that forces `r11` on x86-64 never arises. Safe by circumstance, not by construction.

## Environment

- FEX built from `main` (FEX-2608 + ~130 commits)
- Host: aarch64 Linux (Arch ARM) guest under QEMU on Apple Silicon (M3 Max)
- Guest thunks built for x86-64
- Reproduced with Overwatch 2 under Proton/Wine; DXVK reaches `vkGetQueryPoolResults` during D3D11 device creation and hits an affected instantiation first

## Verification gotcha for reproducers

Do **not** verify with a native `objdump` on an aarch64 host - it cannot disassemble x86-64 guest libraries and exits 0 after a few lines, so `objdump -d | grep fs:0x28` silently reports 0 matches even for known-broken libs. Use `llvm-objdump`. Also, raw `fs:0x28` counts over-report: most hits are ordinary exported pack functions using `r11` as scratch with no `host_addr`. The meaningful metric is a canary load into `r11` *inside* `CallHostFunction`.

Contributor guide

Open the contributing guide

Research direction

Start in ThunkLibs/include/common/Guest.h at CallHostFunction, then inspect the generated x86-64 prologue for affected guest thunk libraries. Use llvm-objdump rather than native objdump and compare the affected instantiations before and after the change. Done means the host callee address remains intact and meaningful canary loads into r11 no longer occur inside CallHostFunction.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, linux
Domain
compilers, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.