FEX-Emu / FEX-Emu/FEX

Async signal ucontext RIP rewrite corrupts guest state

Open
#5,810 1 comment 0 reactions 0 assignees View on GitHub
ml-report
Dominant language
C++
Stars
8k
Forks
351
Avg merge
12h 31m
Merged PRs (30d)
102

Description

### Symptom

amd64 Go under FEX crashes nondeterministically when Go async preemption is enabled. Disabling async preemption avoids the issue:

```sh
FEX ~/opt/go/bin/go test fmt
# crashes / corrupts runtime state

GODEBUG=asyncpreemptoff=1 FEX ~/opt/go/bin/go test fmt
# ok fmt
```

### Minimal non-Go repro

This mimics the key part of Go async preemption: an async `SIGURG` handler rewrites the interrupted `ucontext` RIP to a trampoline, then resumes the original PC.

```c
#define _GNU_SOURCE
#include
#include
#include
#include
#include
#include
#include

extern uint64_t worker(uint64_t n);
extern char worker_lo[], worker_hi[];

asm(".text\n"
".globl worker,worker_lo,worker_hi\n"
"worker:\n"
"worker_lo:\n"
" push %rbx\n"
" mov %rdi, %rcx\n"
" xor %rax, %rax\n"
" xor %rbx, %rbx\n"
" mov $0x9E3779B97F4A7C15, %rsi\n"
"1:\n"
" add $1, %rax\n"
" add %rax, %rbx\n"
" mov %rbx, %rdx\n"
" imul %rsi, %rdx\n"
" xor %rdx, %rax\n"
" rol $7, %rax\n"
" add %rbx, %rax\n"
" sub $1, %rcx\n"
" jnz 1b\n"
" pop %rbx\n"
" ret\n"
"worker_hi:\n");

uintptr_t resume_slot;
extern void trampoline_global(void);
asm(".text\n"
".globl trampoline_global\n"
"trampoline_global:\n"
" jmp *resume_slot(%rip)\n");

static volatile int stop_flag;
static volatile unsigned long preemptions;
static pthread_t worker_tid;

static void handler(int sig, siginfo_t *si, void *uc) {
(void)sig;
(void)si;

ucontext_t *u = (ucontext_t *)uc;
greg_t *g = u->uc_mcontext.gregs;
uintptr_t rip = (uintptr_t)g[REG_RIP];

if (rip < (uintptr_t)worker_lo || rip >= (uintptr_t)worker_hi)
return;

preemptions++;
resume_slot = rip;
g[REG_RIP] = (greg_t)(uintptr_t)trampoline_global;
}

static void *signaller(void *arg) {
(void)arg;

while (!stop_flag) {
pthread_kill(worker_tid, SIGURG);
for (volatile int i = 0; i < 200; i++) {}
}

return NULL;
}

int main(void) {
const uint64_t n = 200000;
const int rounds = 10;

uint64_t expected = worker(n);
printf("expected=0x%016llx\n", (unsigned long long)expected);

struct sigaction sa;
memset(&sa, 0, sizeof(sa));
sa.sa_sigaction = handler;
sa.sa_flags = SA_SIGINFO | SA_RESTART;
sigaction(SIGURG, &sa, NULL);

worker_tid = pthread_self();

pthread_t t;
pthread_create(&t, NULL, signaller, NULL);

int bad = 0;
for (int i = 0; i < rounds; i++) {
uint64_t got = worker(n);
if (got != expected) {
bad++;
printf("mismatch round %d: got=0x%016llx want=0x%016llx\n",
i, (unsigned long long)got, (unsigned long long)expected);
}
}

stop_flag = 1;
pthread_join(t, NULL);

printf("preemptions=%lu mismatches=%d/%d\n", preemptions, bad, rounds);
return bad ? 1 : 0;
}
```

Build/run inside an amd64 guest/rootfs:

```sh
gcc -O2 -pthread repro.c -o repro
FEX ./repro
```

Expected: nonzero `preemptions`, zero mismatches.

Observed under FEX: checksum mismatches, hangs, or crashes. This suggests FEX is exposing an imprecise async guest context when the signal handler rewrites RIP, which matches Go's SIGURG async preemption behavior.

Contributor guide

Open the contributing guide

Research direction

Build and run repro.c with gcc -O2 -pthread, first confirming the nonzero preemptions and zero mismatches expected outside FEX and the corruption observed under FEX. Then trace FEX's SIGURG and ucontext handling, focusing on how rewriting RIP preserves the interrupted guest state; done means the repro and Go's fmt test complete without mismatches, hangs, or crashes.

Written by the indexing model from the issue text.

Assessment

Tech stack
c, cpp, linux
Domain
operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.