FEX-Emu / FEX-Emu/FEX

Blocking NtReadFile freezes all guest compilation once DEP is disabled

Open
#5,907 0 comments 0 reactions 0 assignees View on GitHub
ml-report
Dominant language
C++
Stars
8k
Forks
351
Avg merge
12h 31m
Merged PRs (30d)
102

Description

`BTCpuNotifyReadFile` holds `ThreadCreationMutex` and the process-wide
`CodeInvalidationMutex` — exclusively — for the entire duration of the read it
brackets. Every other `pBTCpuNotify*` hook wraps a memory-management syscall
that returns promptly; `NtReadFile` can block indefinitely. While it does, no
other guest thread in the process can compile code, so any thread that takes a
dispatcher miss deadlocks until the read completes.

Affects both front ends: `Source/Windows/WOW64/Module.cpp` and
`Source/Windows/ARM64EC/Module.cpp`.

### Why it isn't hit constantly

The locked path is only taken when `BeginUntrackedWriteLocked` returns true,
i.e. the read destination overlaps an RWX interval. An ordinary stack buffer
doesn't, so normally both locks are dropped before the read.

That changes when DEP is switched off process-wide. Any 32-bit DLL without
`IMAGE_DLLCHARACTERISTICS_NX_COMPAT` causes Wine's loader to disable DEP, and
`InvalidationTracker::HandleProcessExecuteFlagsChange` then sweeps the whole
address space and promotes **every committed readable+writable region — thread
stacks and heaps included — into `RWXIntervals`**. From that point, a plain
blocking read into a stack buffer is a global compilation barrier.

### Real-world impact

`msiexec /i .msi /q` hangs on packages with 32-bit custom actions.
msiexec's custom-action host (`programs/msiexec/msiexec.c`, `custom_action_server`)
loops on a blocking `ReadFile` of the next action's GUID into a stack local. The
parent is simultaneously in `WaitForSingleObject(thread, INFINITE)` on the action
thread it just handed over, and that thread needs the invalidation mutex shared
for its first dispatcher miss. Nothing proceeds.

Characteristically, the *first* custom action succeeds and the second never does
— the first runs before the non-NX_COMPAT DLL loads and disables DEP.

This isn't MSI-specific: any 32-bit guest that loads such a DLL and then blocks
in a read while another thread runs new code will hang. Older installers and
games commonly do both.

### Minimal reproduction

No MSI needed. A 32-bit guest that:

1. Disables DEP — `NtSetInformationProcess(ProcessExecuteFlags)` with
`MEM_EXECUTE_OPTION_ENABLE` (**0x02**; 0x01 is `_DISABLE`).
2. Creates a pipe with nothing to read.
3. Blocks its main thread in `ReadFile` into a **stack** buffer.
4. Has a second thread execute never-before-run code (e.g. a few `LoadLibraryA`
calls) while that read is outstanding.

With DEP left enabled the worker completes immediately and the read returns on
schedule. With DEP disabled the whole process wedges until the read is
interrupted.

Confirming it's this code path and not something else: `FEX_SMCCHECKS=none`
makes the hang disappear, because `SMCDetectionDisabled` short-circuits
`ProtectRWXIntervalsInternal`, so `BeginUntrackedWriteLocked` returns false and
the locks are released before the read.

### Environment

FEX `a6e74cdb`, arm64 macOS host, Wine 11.16 WoW64. Reproduces on both the
WoW64 and ARM64EC front ends.

### Suggested direction

Two independent points, either of which would fix it:

- Don't hold process-wide locks across `NtReadFile`. Untrap and invalidate
before the syscall, then invalidate the written range afterwards — invalidating
after the write is sound, since it discards anything compiled from a
partially-written buffer. Something is needed to stop a concurrent trap
re-protecting the pages mid-read.
- Don't treat DEP-promoted intervals as RWX for this hook. Those exist because
DEP was disabled, not because the guest declared the memory executable, and
a read destination in one is far more likely a stack buffer than code.

Contributor guide

Open the contributing guide

Research direction

Start by tracing BTCpuNotifyReadFile and BeginUntrackedWriteLocked in Source/Windows/WOW64/Module.cpp and Source/Windows/ARM64EC/Module.cpp, then reproduce with the described blocking pipe read, DEP disabled, and a worker thread. Compare behavior with FEX_SMCCHECKS=none. Done means the blocking read no longer prevents concurrent guest compilation in both front ends, with the regression covered by an appropriate test or reproduction.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
compilers, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.