FEX x86_64 user-mode fails to enumerate .ned files correctly (SPEC CPU2017 620.omnetpp_s)
- Dominant language
- C++
- Stars
- 8k
- Forks
- 351
- Avg merge
- 12h 31m
- Merged PRs (30d)
- 102
Description
## Summary
When running the SPEC CPU2017 benchmark 620.omnetpp_s (OMNeT++ 4.0) on an ARM64 host using FEX (x86_64 user-mode), OMNeT++ fails to load most `.ned` files from a directory, and then reports that the main network `largeNet` is missing. The same x86_64 binary and configuration run correctly under `qemu-x86_64` on the same ARM64 machine.
This looks like a bug in how FEX implements directory enumeration (`readdir`/`getdents`-style syscalls) for x86_64 user-mode.
## Environment
- Host architecture: ARM64
- Host OS: Ubuntu 24.04 (noble) arm64 (in a container)
- FEX: [main branch, commit hash: bf9ab7ffbee9bc14f21741fe265d99f4df9a2680], Rootfs download by FEXRootFSFetcher
- SPEC CPU version: SPEC CPU2017
- Benchmark: 620.omnetpp_s (omnetpp_s_base.x64_static-m64)
- OMNeT++ version: 4.0 (as shipped by SPEC2017)
- QEMU: `qemu-user` package (`qemu-x86_64`)
SPEC run directory (inside container / host FS):
- `/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000`
NED directory:
- `/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned`
`ned/` contains 13 `.ned` files (no `Cli.ned` originally):
- EtherAppCli.ned
- EtherAppSrv.ned
- EtherBus.ned
- EtherEncap.ned
- EtherHost.ned
- EtherHub.ned
- EtherLLC.ned
- EtherMAC.ned
- EtherSwitch.ned
- LargeNet.ned
- MACRelayUnit.ned
- MACRelayUnitNP.ned
- MACRelayUnitPP.ned
Both on the host and inside `FEX /bin/bash`, `ls` shows the same 13 files.
## Reproduction
From the SPEC run directory:
```sh
cd /host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000
```
### 1. Run with qemu-x86_64 (works)
```sh
qemu-x86_64 ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0
```
Output (abridged):
- `Loading NED files from a SPEC directory 14`
- `Setting up network 'largeNet'...`
- Simulation runs normally
### 2. Run with FEX (fails)
```sh
FEX ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0
```
Without any manual changes, OMNeT++ prints:
```text
Error: Error loading NED sources from `/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned': cannot stat file 'Cli.ned': No such file or directory.
```
Note: there is no `Cli.ned` in the `ned/` directory; SPEC’s reference x86_64 run on a native x86_64 machine also works fine without such a file.
If I manually create an empty `Cli.ned`:
```sh
cd ned
touch Cli.ned
cd ..
```
and re-run under FEX:
```sh
FEX ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0
```
I get a bit further:
```text
OMNeT++ Discrete Event Simulation (C) 1992-2008 Andras Varga, OpenSim Ltd.
Version: 4.0, build: 090310-10709, edition: Academic Public License -- NOT FOR COMMERCIAL USE
Setting up Cmdenv...
Loading NED files from a SPEC directory 2
Preparing for running configuration General, run #0...
Scenario: $repetition=0
Assigned runID=speccpu-runid
Error: Network `largeNet' not found, check .ini and .ned files.
End.
```
Key difference:
- Under **qemu-x86_64**, OMNeT++ reports: `Loading NED files from a SPEC directory 14`
- Under **FEX**, OMNeT++ reports: `Loading NED files from a SPEC directory 2`, and then fails with `Network 'largeNet' not found`
The only difference between the runs is the x86_64 emulator (qemu vs FEX); same working directory, same `omnetpp.ini`, same binary.
## File system visibility check
On host / container:
```sh
cd /host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000
ls -l ned/ | wc -l
ls -l ned/
```
Output shows 13 `.ned` files as listed above.
Inside `FEX /bin/bash`:
```sh
FEX /bin/bash
cd /host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000
ls -l ned/ | wc -l
ls -l ned/
```
Output is identical: same 13 `.ned` files with same sizes and permissions. So the host FS is mapped correctly into FEX (at least for shell utilities).
## strace comparison (qemu-x86_64 vs FEX)
I captured `strace` logs for both qemu-x86_64 and FEX runs.
### qemu-x86_64
Command:
```sh
strace -f -o qemu.log \
qemu-x86_64 ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0
```
From `qemu.log`, OMNeT++ clearly enumerates and opens many `.ned` files in `ned/`:
```text
newfstatat(AT_FDCWD, "EtherAppCli.ned", {st_mode=S_IFREG|0775, st_size=1678, ...}, 0) = 0
readlinkat(AT_FDCWD, "/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/EtherAppCli.ned", 0xffffed9884f0, 1023) = -1 EINVAL (Invalid argument)
faccessat(AT_FDCWD, "/usr/gnemul/qemu-x86_64/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/EtherAppCli.ned", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/EtherAppCli.ned", O_RDONLY) = 4
newfstatat(AT_FDCWD, "EtherAppSrv.ned", {st_mode=S_IFREG|0775, st_size=1137, ...}, 0) = 0
openat(AT_FDCWD, "/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/EtherAppSrv.ned", O_RDONLY) = 4
newfstatat(AT_FDCWD, "EtherBus.ned", {st_mode=S_IFREG|0775, st_size=1843, ...}, 0) = 0
...
(newfstatat/openat for many other .ned files, including LargeNet.ned, MACRelayUnit*.ned, etc.)
```
So under qemu-x86_64 the x86_64 process successfully enumerates and opens all `.ned` files in the directory.
### FEX
Command:
```sh
strace -f -o fex.log \
FEX ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0
```
From `fex.log` (without my manually created empty Cli.ned):
```text
openat2(5, "host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/package.ned", {flags=O_RDONLY, resolve=RESOLVE_IN_ROOT}, 24) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/package.ned", O_RDONLY) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "Cli.ned", 0xfffffc18a080, 0) = -1 ENOENT (No such file or directory)
```
Notably:
- There are **no** `newfstatat/openat` calls for the existing `.ned` files such as `EtherAppCli.ned`, `LargeNet.ned`, etc.
- Only `package.ned` and `Cli.ned` are ever attempted.
- This is consistent with OMNeT++ reporting `Loading NED files from a SPEC directory 2` under FEX vs `14` under qemu.
So although a normal shell inside FEX can `ls` all files in `ned/`, the x86_64 process under FEX apparently **does not see / enumerate those entries** when scanning for NED files.
This strongly suggests an issue in FEX’s implementation of directory enumeration syscalls (`getdents`, `getdents64`, or related) or path resolution for the emulated x86_64 process.
## Workarounds
I found two workarounds:
1. **Manually creating an empty `Cli.ned`**
This bypasses the initial `cannot stat file 'Cli.ned'` error but still fails later with `Network 'largeNet' not found`, because most `.ned` files are not being loaded.
2. **Explicitly specifying all `.ned` files in `omnetpp.ini`**
If I list all `.ned` files explicitly via `ned-files = ned/EtherAppCli.ned ned/EtherAppSrv.ned ...` in `omnetpp.ini`, OMNeT++ can open them directly (without relying on directory enumeration), and the simulation can run further under FEX.
In contrast, `qemu-x86_64` works out of the box, without any changes to the SPEC-supplied configuration, which matches behavior on a native x86_64 host.
## Expected behavior
- The x86_64 process running under FEX should see the same directory entries as under qemu-x86_64 / native x86_64.
- OMNeT++ should be able to enumerate and open all `.ned` files under `ned/`, resulting in `Loading NED files from a SPEC directory 14`, and the network `largeNet` should be found and the simulation should run normally.
## Actual behavior
- Under FEX, OMNeT++ only sees 2 NED sources (vs 14 under qemu).
- It attempts to access `package.ned` and `Cli.ned` but never attempts to `stat/open` real existing NED files like `EtherAppCli.ned`, `LargeNet.ned`, etc.
- As a result, it fails with:
```text
Error: Network `largeNet' not found, check .ini and .ned files.
```
## Request
Could you please:
1. Confirm whether this is a known issue with x86_64 user-mode directory enumeration in FEX?
2. Suggest a proper fix or workaround (other than manually listing all `.ned` files)?
3. If helpful, I can provide the full `qemu.log` and `fex.log` files or further reduced repros.
Thanks!
Contributor guide
Assessment
This issue has not been assessed yet.