FEX-Emu / FEX-Emu/FEX

FEXOfflineCompiler exits silently on Windows: null MetaLayer deref in main()

Open
#5,817 0 comments 0 reactions 0 assignees View on GitHub
ml-report
Dominant language
C++
Stars
8k
Forks
351
Avg merge
12h 31m
Merged PRs (30d)
102

Description

`FEXOfflineCompiler`'s Windows startup path faults on the first statement of
`main()`, exiting without printing anything — not even its own usage text.

**Upfront about what I have and have not tested:** I observed this under
FEX/ARM64EC on macOS via Wine, running the Windows PE build — not on real
Windows. The faulting code is in the Windows-only branch and the defect is
visible by inspection, which is why the case below is built from source rather
than from my repro. I cannot claim to have watched it fail on Windows itself.

This is a null dereference by inspection, so it can be confirmed from the source
without reproducing anything:

- `main()` calls `FEX::Windows::Logging::Init()` as its first statement
([`Main.cpp:850`](https://github.com/FEX-Emu/FEX/blob/9686454/Source/Tools/FEXOfflineCompiler/Main.cpp#L850)).
- `Init()` immediately reads `SILENTLOG`
([`Logging.cpp:36`](https://github.com/FEX-Emu/FEX/blob/9686454/Source/Windows/Common/Logging.cpp#L36)).
- That read reaches `FEXCore::Config::GetConv`, which dereferences `Meta`
unconditionally
([`Config.cpp:433`](https://github.com/FEX-Emu/FEX/blob/9686454/FEXCore/Source/Interface/Config/Config.cpp#L433)).
- `Meta` is a null-initialised static
([`:116`](https://github.com/FEX-Emu/FEX/blob/9686454/FEXCore/Source/Interface/Config/Config.cpp#L116))
assigned only in `Initialize()`
([`:239`](https://github.com/FEX-Emu/FEX/blob/9686454/FEXCore/Source/Interface/Config/Config.cpp#L239)),
which has not run yet.
- The fault materialises one frame further in, at `MetaLayer::GetConv`'s first
statement — `OptionMap.find(Option)`
([`:140`](https://github.com/FEX-Emu/FEX/blob/9686454/FEXCore/Source/Interface/Config/Config.cpp#L140)) — a member access
through a null `this`.

Nothing loads config before that point. `FEX::Config::LoadConfig` is first reached
inside `GenerateCache`
([`Main.cpp:615`](https://github.com/FEX-Emu/FEX/blob/9686454/Source/Tools/FEXOfflineCompiler/Main.cpp#L615)),
well after `Logging::Init()`, and `ProcessAll()` never calls it at all. Since the
fault is in `main()` itself, neither subcommand is reachable — including the
`generate` child that `ProcessAll()` would otherwise spawn on Windows
([`Main.cpp:834`](https://github.com/FEX-Emu/FEX/blob/9686454/Source/Tools/FEXOfflineCompiler/Main.cpp#L834)),
which would hit the same fault at startup. Fixing `main()` covers both.

**Both other Windows entry points already order this correctly:**

| Entry point | Order of calls |
|---|---|
| [`ARM64EC/Module.cpp:585-587`](https://github.com/FEX-Emu/FEX/blob/9686454/Source/Windows/ARM64EC/Module.cpp#L585-L587) | `LoadConfig` → `ReloadMetaLayer` → `Logging::Init` |
| [`WOW64/Module.cpp:519-521`](https://github.com/FEX-Emu/FEX/blob/9686454/Source/Windows/WOW64/Module.cpp#L519-L521) | same |
| [`FEXOfflineCompiler/Main.cpp:850`](https://github.com/FEX-Emu/FEX/blob/9686454/Source/Tools/FEXOfflineCompiler/Main.cpp#L850) | `Logging::Init` only |

Adding the same `LoadConfig` + `ReloadMetaLayer` pair before `Logging::Init()`
fixes it.

Note the call sits under a plain `#ifndef _WIN32 / #else`, so this affects every
Windows build, not only ARM64EC.

## What I actually ran, and what I could not

Being precise about this, because it limits what I can claim.

I could **not** run the binary under its installed name. On my setup any PE with a
filename of 20 characters or more faults before entry — a separate, macOS-side
problem I have not root-caused and am not reporting here.
`FEXOfflineCompiler64.exe` is 24 characters, so it never starts for me for a
reason that has nothing to do with this bug. What I ran was the same binary
renamed:

```
fexofc.exe # 10 chars, so the name-length problem does not apply
```

- **Expected:** the usage block, and exit 1.
- **Actual:** exits immediately with no output at all.

Any argument behaves the same; the fault is before argument parsing. Debugging
that silent exit is what produced the analysis above — the fault is in
`MetaLayer::GetConv`, reached from `Logging::Init()`.

Why that produces no output at all here, rather than an unhandled-exception
message, I have not established; it is not part of the claim above.

**The honest limitation:** at the stock name I cannot reach this fault at all —
the name-length problem fires first, before entry — so the configuration you
actually ship is untestable on my machine, and the short name is what makes this
one reachable. The argument above therefore rests on source inspection rather than
on my repro: the ordering in `main()` is wrong regardless of filename, and that is
checkable in your tree without running anything.

If someone with Windows-on-ARM hardware runs `FEXOfflineCompiler64.exe` with no
arguments, that settles it in ten seconds.

## Secondary: `size_t` underflow in the sibling-binary path

Unrelated to the above, and latent rather than blocking, but it is a one-line
fragility in the same file
([`Main.cpp:789`](https://github.com/FEX-Emu/FEX/blob/9686454/Source/Tools/FEXOfflineCompiler/Main.cpp#L789)):

```cpp
auto NewExecName = fmt::format("FEXOfflineCompiler{}.exe", ExecutableIt->second.ExecutableBitness.value());
SelfPath.replace(SelfPath.size() - NewExecName.size(), NewExecName.size(), NewExecName);
```

`NewExecName` is 24 characters. If `SelfPath` is shorter, the subtraction
underflows and `replace` throws `std::out_of_range`, after the code maps have
already been aggregated. It also assumes the binary is always invoked as
`FEXOfflineCompilerNN.exe`; under any other name the replacement silently corrupts
the path instead of pointing at a sibling. Computing the last path separator and
replacing only the filename component avoids both.

**How I hit it.** With the config-ordering fix above applied locally — without it
the tool never starts, so this is unreachable — and the binary renamed to
`fexofc.exe` for the platform reason above, `process-all` aggregated the code maps
([`Main.cpp:765`](https://github.com/FEX-Emu/FEX/blob/9686454/Source/Tools/FEXOfflineCompiler/Main.cpp#L765))
and then threw `std::out_of_range` on the first executable entry it found, at
[`Main.cpp:789`](https://github.com/FEX-Emu/FEX/blob/9686454/Source/Tools/FEXOfflineCompiler/Main.cpp#L789):
`SelfPath` was `C:\fexofc.exe`, shorter than the 24-character name the code assumes
it still has. That is before it prints `Checking caches for executable`
([`:792`](https://github.com/FEX-Emu/FEX/blob/9686454/Source/Tools/FEXOfflineCompiler/Main.cpp#L792)),
and well before the `_spawnv` the path is being built for
([`:834`](https://github.com/FEX-Emu/FEX/blob/9686454/Source/Tools/FEXOfflineCompiler/Main.cpp#L834)).

To be clear on the order: the rename works around a quirk of my platform, not the
config-ordering defect — that one needed the source change above. What the rename
does is expose this second defect. Renaming a binary is an ordinary thing to do,
and I would not expect the tool to assume its own installed filename — which is the
actual fix here, independent of my platform.

At the stock name and install path this will not fire, so it is latent upstream
rather than blocking. Raising it as a fragility worth hardening, not as something
that breaks a supported configuration.

## Notes

- **I used an AI assistant (Claude Code) during this investigation.** Per
`CONTRIBUTING.md` ("No AI/ML/LLM/etc code contributions") I am therefore not
opening a PR — I would rather you write anything that lands in the tree.
- What I changed locally is here for checking, not for merging:
[`tompollok/FEX@7fb37b1`](https://github.com/tompollok/FEX/commit/7fb37b1), a
single commit touching only `Main.cpp` on top of
[`9686454`](https://github.com/FEX-Emu/FEX/commit/9686454).
- Runs were on `9686454` plus two local commits: macOS build shims for the ARM64EC
unix library (unrelated to this report) and the ordering change linked above.
ARM64EC, macOS 26.6 (Apple M3 Max), Wine 11.14, llvm-mingw ARM64EC toolchain.
The analysis itself is against stock `9686454`.

Happy to test a patch and report back.

Contributor guide

Open the contributing guide

Research direction

Start in Source/Tools/FEXOfflineCompiler/Main.cpp at main(), GenerateCache(), ProcessAll(), and the sibling-binary path around lines 789-834. Compare the Windows startup ordering with Source/Windows/ARM64EC/Module.cpp and WOW64/Module.cpp, then inspect the referenced Config.cpp and Logging.cpp paths. Done means the Windows tool reaches usage and both subcommands, while renamed invocations preserve a valid sibling executable path.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cli, operating-systems, tooling
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
67/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.