Blocking NtReadFile freezes all guest compilation once DEP is disabled
- Dominant language
- C++
- Stars
- 8k
- Forks
- 351
- Avg merge
- 12h 31m
- Merged PRs (30d)
- 102
Description
`BTCpuNotifyReadFile` holds `ThreadCreationMutex` and the process-wide
`CodeInvalidationMutex` — exclusively — for the entire duration of the read it
brackets. Every other `pBTCpuNotify*` hook wraps a memory-management syscall
that returns promptly; `NtReadFile` can block indefinitely. While it does, no
other guest thread in the process can compile code, so any thread that takes a
dispatcher miss deadlocks until the read completes.
Affects both front ends: `Source/Windows/WOW64/Module.cpp` and
`Source/Windows/ARM64EC/Module.cpp`.
### Why it isn't hit constantly
The locked path is only taken when `BeginUntrackedWriteLocked` returns true,
i.e. the read destination overlaps an RWX interval. An ordinary stack buffer
doesn't, so normally both locks are dropped before the read.
That changes when DEP is switched off process-wide. Any 32-bit DLL without
`IMAGE_DLLCHARACTERISTICS_NX_COMPAT` causes Wine's loader to disable DEP, and
`InvalidationTracker::HandleProcessExecuteFlagsChange` then sweeps the whole
address space and promotes **every committed readable+writable region — thread
stacks and heaps included — into `RWXIntervals`**. From that point, a plain
blocking read into a stack buffer is a global compilation barrier.
### Real-world impact
`msiexec /i .msi /q` hangs on packages with 32-bit custom actions.
msiexec's custom-action host (`programs/msiexec/msiexec.c`, `custom_action_server`)
loops on a blocking `ReadFile` of the next action's GUID into a stack local. The
parent is simultaneously in `WaitForSingleObject(thread, INFINITE)` on the action
thread it just handed over, and that thread needs the invalidation mutex shared
for its first dispatcher miss. Nothing proceeds.
Characteristically, the *first* custom action succeeds and the second never does
— the first runs before the non-NX_COMPAT DLL loads and disables DEP.
This isn't MSI-specific: any 32-bit guest that loads such a DLL and then blocks
in a read while another thread runs new code will hang. Older installers and
games commonly do both.
### Minimal reproduction
No MSI needed. A 32-bit guest that:
1. Disables DEP — `NtSetInformationProcess(ProcessExecuteFlags)` with
`MEM_EXECUTE_OPTION_ENABLE` (**0x02**; 0x01 is `_DISABLE`).
2. Creates a pipe with nothing to read.
3. Blocks its main thread in `ReadFile` into a **stack** buffer.
4. Has a second thread execute never-before-run code (e.g. a few `LoadLibraryA`
calls) while that read is outstanding.
With DEP left enabled the worker completes immediately and the read returns on
schedule. With DEP disabled the whole process wedges until the read is
interrupted.
Confirming it's this code path and not something else: `FEX_SMCCHECKS=none`
makes the hang disappear, because `SMCDetectionDisabled` short-circuits
`ProtectRWXIntervalsInternal`, so `BeginUntrackedWriteLocked` returns false and
the locks are released before the read.
### Environment
FEX `a6e74cdb`, arm64 macOS host, Wine 11.16 WoW64. Reproduces on both the
WoW64 and ARM64EC front ends.
### Suggested direction
Two independent points, either of which would fix it:
- Don't hold process-wide locks across `NtReadFile`. Untrap and invalidate
before the syscall, then invalidate the written range afterwards — invalidating
after the write is sound, since it discards anything compiled from a
partially-written buffer. Something is needed to stop a concurrent trap
re-protecting the pages mid-read.
- Don't treat DEP-promoted intervals as RWX for this hook. Those exist because
DEP was disabled, not because the guest declared the memory executable, and
a read destination in one is far more likely a stack buffer than code.
Contributor guide
Research direction
Start by tracing BTCpuNotifyReadFile and BeginUntrackedWriteLocked in Source/Windows/WOW64/Module.cpp and Source/Windows/ARM64EC/Module.cpp, then reproduce with the described blocking pipe read, DEP disabled, and a worker thread. Compare behavior with FEX_SMCCHECKS=none. Done means the blocking read no longer prevents concurrent guest compilation in both front ends, with the regression covered by an appropriate test or reproduction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- compilers, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100