FEX-Emu / FEX-Emu/FEX

FEX x86_64 user-mode fails to enumerate .ned files correctly (SPEC CPU2017 620.omnetpp_s)

Open
#5,115 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
8k
Forks
351
Avg merge
12h 31m
Merged PRs (30d)
102

Description

## Summary

When running the SPEC CPU2017 benchmark 620.omnetpp_s (OMNeT++ 4.0) on an ARM64 host using FEX (x86_64 user-mode), OMNeT++ fails to load most `.ned` files from a directory, and then reports that the main network `largeNet` is missing. The same x86_64 binary and configuration run correctly under `qemu-x86_64` on the same ARM64 machine.

This looks like a bug in how FEX implements directory enumeration (`readdir`/`getdents`-style syscalls) for x86_64 user-mode.

## Environment

- Host architecture: ARM64
- Host OS: Ubuntu 24.04 (noble) arm64 (in a container)
- FEX: [main branch, commit hash: bf9ab7ffbee9bc14f21741fe265d99f4df9a2680], Rootfs download by FEXRootFSFetcher
- SPEC CPU version: SPEC CPU2017
- Benchmark: 620.omnetpp_s (omnetpp_s_base.x64_static-m64)
- OMNeT++ version: 4.0 (as shipped by SPEC2017)
- QEMU: `qemu-user` package (`qemu-x86_64`)

SPEC run directory (inside container / host FS):

- `/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000`

NED directory:

- `/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned`

`ned/` contains 13 `.ned` files (no `Cli.ned` originally):

- EtherAppCli.ned
- EtherAppSrv.ned
- EtherBus.ned
- EtherEncap.ned
- EtherHost.ned
- EtherHub.ned
- EtherLLC.ned
- EtherMAC.ned
- EtherSwitch.ned
- LargeNet.ned
- MACRelayUnit.ned
- MACRelayUnitNP.ned
- MACRelayUnitPP.ned

Both on the host and inside `FEX /bin/bash`, `ls` shows the same 13 files.

## Reproduction

From the SPEC run directory:

```sh
cd /host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000
```

### 1. Run with qemu-x86_64 (works)

```sh
qemu-x86_64 ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0
```

Output (abridged):

- `Loading NED files from a SPEC directory 14`
- `Setting up network 'largeNet'...`
- Simulation runs normally

### 2. Run with FEX (fails)

```sh
FEX ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0
```

Without any manual changes, OMNeT++ prints:

```text
Error: Error loading NED sources from `/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned': cannot stat file 'Cli.ned': No such file or directory.
```

Note: there is no `Cli.ned` in the `ned/` directory; SPEC’s reference x86_64 run on a native x86_64 machine also works fine without such a file.

If I manually create an empty `Cli.ned`:

```sh
cd ned
touch Cli.ned
cd ..
```

and re-run under FEX:

```sh
FEX ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0
```

I get a bit further:

```text
OMNeT++ Discrete Event Simulation (C) 1992-2008 Andras Varga, OpenSim Ltd.
Version: 4.0, build: 090310-10709, edition: Academic Public License -- NOT FOR COMMERCIAL USE
Setting up Cmdenv...
Loading NED files from a SPEC directory 2

Preparing for running configuration General, run #0...
Scenario: $repetition=0
Assigned runID=speccpu-runid

Error: Network `largeNet' not found, check .ini and .ned files.

End.
```

Key difference:

- Under **qemu-x86_64**, OMNeT++ reports: `Loading NED files from a SPEC directory 14`
- Under **FEX**, OMNeT++ reports: `Loading NED files from a SPEC directory 2`, and then fails with `Network 'largeNet' not found`

The only difference between the runs is the x86_64 emulator (qemu vs FEX); same working directory, same `omnetpp.ini`, same binary.

## File system visibility check

On host / container:

```sh
cd /host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000
ls -l ned/ | wc -l
ls -l ned/
```

Output shows 13 `.ned` files as listed above.

Inside `FEX /bin/bash`:

```sh
FEX /bin/bash
cd /host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000
ls -l ned/ | wc -l
ls -l ned/
```

Output is identical: same 13 `.ned` files with same sizes and permissions. So the host FS is mapped correctly into FEX (at least for shell utilities).

## strace comparison (qemu-x86_64 vs FEX)

I captured `strace` logs for both qemu-x86_64 and FEX runs.

### qemu-x86_64

Command:

```sh
strace -f -o qemu.log \
qemu-x86_64 ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0
```

From `qemu.log`, OMNeT++ clearly enumerates and opens many `.ned` files in `ned/`:

```text
newfstatat(AT_FDCWD, "EtherAppCli.ned", {st_mode=S_IFREG|0775, st_size=1678, ...}, 0) = 0
readlinkat(AT_FDCWD, "/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/EtherAppCli.ned", 0xffffed9884f0, 1023) = -1 EINVAL (Invalid argument)
faccessat(AT_FDCWD, "/usr/gnemul/qemu-x86_64/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/EtherAppCli.ned", F_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/EtherAppCli.ned", O_RDONLY) = 4

newfstatat(AT_FDCWD, "EtherAppSrv.ned", {st_mode=S_IFREG|0775, st_size=1137, ...}, 0) = 0
openat(AT_FDCWD, "/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/EtherAppSrv.ned", O_RDONLY) = 4

newfstatat(AT_FDCWD, "EtherBus.ned", {st_mode=S_IFREG|0775, st_size=1843, ...}, 0) = 0
...

(newfstatat/openat for many other .ned files, including LargeNet.ned, MACRelayUnit*.ned, etc.)
```

So under qemu-x86_64 the x86_64 process successfully enumerates and opens all `.ned` files in the directory.

### FEX

Command:

```sh
strace -f -o fex.log \
FEX ../../exe/omnetpp_s_base.x64_static-m64 -f ./omnetpp.ini -c General -r 0
```

From `fex.log` (without my manually created empty Cli.ned):

```text
openat2(5, "host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/package.ned", {flags=O_RDONLY, resolve=RESOLVE_IN_ROOT}, 24) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/host/SPEC2017/620.omnetpp_s/run/run_base_refspeed_x64_static-m64.0000/ned/package.ned", O_RDONLY) = -1 ENOENT (No such file or directory)
newfstatat(AT_FDCWD, "Cli.ned", 0xfffffc18a080, 0) = -1 ENOENT (No such file or directory)
```

Notably:

- There are **no** `newfstatat/openat` calls for the existing `.ned` files such as `EtherAppCli.ned`, `LargeNet.ned`, etc.
- Only `package.ned` and `Cli.ned` are ever attempted.
- This is consistent with OMNeT++ reporting `Loading NED files from a SPEC directory 2` under FEX vs `14` under qemu.

So although a normal shell inside FEX can `ls` all files in `ned/`, the x86_64 process under FEX apparently **does not see / enumerate those entries** when scanning for NED files.

This strongly suggests an issue in FEX’s implementation of directory enumeration syscalls (`getdents`, `getdents64`, or related) or path resolution for the emulated x86_64 process.

## Workarounds

I found two workarounds:

1. **Manually creating an empty `Cli.ned`**
This bypasses the initial `cannot stat file 'Cli.ned'` error but still fails later with `Network 'largeNet' not found`, because most `.ned` files are not being loaded.

2. **Explicitly specifying all `.ned` files in `omnetpp.ini`**
If I list all `.ned` files explicitly via `ned-files = ned/EtherAppCli.ned ned/EtherAppSrv.ned ...` in `omnetpp.ini`, OMNeT++ can open them directly (without relying on directory enumeration), and the simulation can run further under FEX.

In contrast, `qemu-x86_64` works out of the box, without any changes to the SPEC-supplied configuration, which matches behavior on a native x86_64 host.

## Expected behavior

- The x86_64 process running under FEX should see the same directory entries as under qemu-x86_64 / native x86_64.
- OMNeT++ should be able to enumerate and open all `.ned` files under `ned/`, resulting in `Loading NED files from a SPEC directory 14`, and the network `largeNet` should be found and the simulation should run normally.

## Actual behavior

- Under FEX, OMNeT++ only sees 2 NED sources (vs 14 under qemu).
- It attempts to access `package.ned` and `Cli.ned` but never attempts to `stat/open` real existing NED files like `EtherAppCli.ned`, `LargeNet.ned`, etc.
- As a result, it fails with:

```text
Error: Network `largeNet' not found, check .ini and .ned files.
```

## Request

Could you please:

1. Confirm whether this is a known issue with x86_64 user-mode directory enumeration in FEX?
2. Suggest a proper fix or workaround (other than manually listing all `.ned` files)?
3. If helpful, I can provide the full `qemu.log` and `fex.log` files or further reduced repros.

Thanks!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.