Async signal ucontext RIP rewrite corrupts guest state
- Dominant language
- C++
- Stars
- 8k
- Forks
- 351
- Avg merge
- 12h 31m
- Merged PRs (30d)
- 102
Description
### Symptom
amd64 Go under FEX crashes nondeterministically when Go async preemption is enabled. Disabling async preemption avoids the issue:
```sh
FEX ~/opt/go/bin/go test fmt
# crashes / corrupts runtime state
GODEBUG=asyncpreemptoff=1 FEX ~/opt/go/bin/go test fmt
# ok fmt
```
### Minimal non-Go repro
This mimics the key part of Go async preemption: an async `SIGURG` handler rewrites the interrupted `ucontext` RIP to a trampoline, then resumes the original PC.
```c
#define _GNU_SOURCE
#include
#include
#include
#include
#include
#include
#include
extern uint64_t worker(uint64_t n);
extern char worker_lo[], worker_hi[];
asm(".text\n"
".globl worker,worker_lo,worker_hi\n"
"worker:\n"
"worker_lo:\n"
" push %rbx\n"
" mov %rdi, %rcx\n"
" xor %rax, %rax\n"
" xor %rbx, %rbx\n"
" mov $0x9E3779B97F4A7C15, %rsi\n"
"1:\n"
" add $1, %rax\n"
" add %rax, %rbx\n"
" mov %rbx, %rdx\n"
" imul %rsi, %rdx\n"
" xor %rdx, %rax\n"
" rol $7, %rax\n"
" add %rbx, %rax\n"
" sub $1, %rcx\n"
" jnz 1b\n"
" pop %rbx\n"
" ret\n"
"worker_hi:\n");
uintptr_t resume_slot;
extern void trampoline_global(void);
asm(".text\n"
".globl trampoline_global\n"
"trampoline_global:\n"
" jmp *resume_slot(%rip)\n");
static volatile int stop_flag;
static volatile unsigned long preemptions;
static pthread_t worker_tid;
static void handler(int sig, siginfo_t *si, void *uc) {
(void)sig;
(void)si;
ucontext_t *u = (ucontext_t *)uc;
greg_t *g = u->uc_mcontext.gregs;
uintptr_t rip = (uintptr_t)g[REG_RIP];
if (rip < (uintptr_t)worker_lo || rip >= (uintptr_t)worker_hi)
return;
preemptions++;
resume_slot = rip;
g[REG_RIP] = (greg_t)(uintptr_t)trampoline_global;
}
static void *signaller(void *arg) {
(void)arg;
while (!stop_flag) {
pthread_kill(worker_tid, SIGURG);
for (volatile int i = 0; i < 200; i++) {}
}
return NULL;
}
int main(void) {
const uint64_t n = 200000;
const int rounds = 10;
uint64_t expected = worker(n);
printf("expected=0x%016llx\n", (unsigned long long)expected);
struct sigaction sa;
memset(&sa, 0, sizeof(sa));
sa.sa_sigaction = handler;
sa.sa_flags = SA_SIGINFO | SA_RESTART;
sigaction(SIGURG, &sa, NULL);
worker_tid = pthread_self();
pthread_t t;
pthread_create(&t, NULL, signaller, NULL);
int bad = 0;
for (int i = 0; i < rounds; i++) {
uint64_t got = worker(n);
if (got != expected) {
bad++;
printf("mismatch round %d: got=0x%016llx want=0x%016llx\n",
i, (unsigned long long)got, (unsigned long long)expected);
}
}
stop_flag = 1;
pthread_join(t, NULL);
printf("preemptions=%lu mismatches=%d/%d\n", preemptions, bad, rounds);
return bad ? 1 : 0;
}
```
Build/run inside an amd64 guest/rootfs:
```sh
gcc -O2 -pthread repro.c -o repro
FEX ./repro
```
Expected: nonzero `preemptions`, zero mismatches.
Observed under FEX: checksum mismatches, hangs, or crashes. This suggests FEX is exposing an imprecise async guest context when the signal handler rewrites RIP, which matches Go's SIGURG async preemption behavior.
Contributor guide
Research direction
Build and run repro.c with gcc -O2 -pthread, first confirming the nonzero preemptions and zero mismatches expected outside FEX and the corruption observed under FEX. Then trace FEX's SIGURG and ucontext handling, focusing on how rewriting RIP preserves the interrupted guest state; done means the repro and Go's fmt test complete without mismatches, hangs, or crashes.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- c, cpp, linux
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100