HotSpot ZGC under FEX corrupts object fields or crashes in ZGC stack/root processing
- Dominant language
- C++
- Stars
- 8k
- Forks
- 351
- Avg merge
- 12h 31m
- Merged PRs (30d)
- 102
Description
### Symptom
Running x86-64 HotSpot with explicit `-XX:+UseZGC` under FEX on an AArch64 host fails quickly on a small Java allocation/GC stress test.
The most common failure is an impossible Java exception:
```text
java.lang.NullPointerException: Cannot load from byte/boolean array because "this.bytes" is null
at FEXZGCBarrierRepro$Payload.sample(FEXZGCBarrierRepro.java:28)
at FEXZGCBarrierRepro.worker(FEXZGCBarrierRepro.java:73)
```
The `bytes` field is `final` and is initialized immediately before the failing load. In other runs, the same test aborts the JVM with a ZGC-related fatal SIGSEGV, for example:
```text
# SIGSEGV (0xb) at pc=0x00007ffff75888ca
# Problematic frame:
# V [libjvm.so+0x13888ca] ZUncoloredRoot::process(zaddress_unsafe*, unsigned long)+0xba
```
### Environment
```text
Host: CentOS Stream 10 aarch64 on Apple Silicon
Kernel: 6.12.0-254.el10.aarch64
FEX commit: c171c0619207fe9316d107c3e6b839fb3db8554f
Guest JDK: Temurin OpenJDK 26.0.2+10, linux x86-64
```
The guest `java` binary is x86-64:
```text
ELF 64-bit LSB pie executable, x86-64, interpreter /lib64/ld-linux-x86-64.so.2
```
### Reproducer
Compile with an x86-64 JDK and run through FEX:
```sh
/path/to/FEX jdk-26.0.2+10/bin/javac FEXZGCBarrierRepro.java
/path/to/FEX jdk-26.0.2+10/bin/java \
-Xms128m -Xmx128m -XX:+UseZGC \
FEXZGCBarrierRepro
```
`FEXZGCBarrierRepro.java`:
```java
import java.util.concurrent.CountDownLatch;
public final class FEXZGCBarrierRepro {
private static final int WORKERS = 4;
private static final int ITERATIONS = 450_000;
private static final int ROOT_SET_SIZE = 4_096;
private static final int GC_REQUESTS = 160;
private static final Object[][] roots = new Object[WORKERS][];
private static volatile Object gcRoot;
private static long checksum;
private FEXZGCBarrierRepro() {
}
private static final class Payload {
private final byte[] bytes;
private final int seed;
private Payload(int seed) {
this.seed = seed;
bytes = new byte[128 + (seed & 255)];
bytes[0] = (byte) seed;
bytes[bytes.length - 1] = (byte) (seed >>> 8);
}
private int sample() {
return mix(seed, bytes[0], bytes[bytes.length - 1]);
}
}
private static int mix(int seed, int first, int last) {
return seed ^ first ^ last;
}
private static long expectedChecksum() {
long total = 0;
for (int worker = 0; worker < WORKERS; worker++) {
for (int i = 0; i < ITERATIONS; i++) {
int seed = (worker << 24) ^ i;
total += mix(seed, (byte) seed, (byte) (seed >>> 8));
}
}
return total;
}
private static void await(CountDownLatch start) {
try {
start.await();
} catch (InterruptedException exception) {
Thread.currentThread().interrupt();
throw new RuntimeException(exception);
}
}
private static void sleepBriefly() {
try {
Thread.sleep(1);
} catch (InterruptedException exception) {
Thread.currentThread().interrupt();
throw new RuntimeException(exception);
}
}
private static void worker(int id, CountDownLatch start) {
await(start);
Object[] localRoots = new Object[ROOT_SET_SIZE];
long localChecksum = 0;
for (int i = 0; i < ITERATIONS; i++) {
Payload payload = new Payload((id << 24) ^ i);
localRoots[i & (ROOT_SET_SIZE - 1)] = payload;
localChecksum += payload.sample();
}
roots[id] = localRoots;
synchronized (FEXZGCBarrierRepro.class) {
checksum += localChecksum;
}
}
private static void requestGc(CountDownLatch start) {
await(start);
for (int i = 0; i < GC_REQUESTS; i++) {
byte[][] batch = new byte[32][];
for (int j = 0; j < batch.length; j++) {
batch[j] = new byte[32 * 1_024 + ((i + j) & 1_023)];
batch[j][0] = (byte) (i ^ j);
}
gcRoot = batch;
System.gc();
sleepBriefly();
}
}
public static void main(String[] args) throws Exception {
Thread.setDefaultUncaughtExceptionHandler((thread, exception) -> {
System.err.println("uncaught exception in " + thread.getName());
exception.printStackTrace();
Runtime.getRuntime().halt(100);
});
CountDownLatch start = new CountDownLatch(1);
Thread[] threads = new Thread[WORKERS];
for (int i = 0; i < threads.length; i++) {
final int id = i;
threads[i] = new Thread(() -> worker(id, start), "worker-" + id);
threads[i].start();
}
Thread gcThread = new Thread(() -> requestGc(start), "gc-requester");
gcThread.start();
start.countDown();
for (Thread thread : threads) {
thread.join();
}
gcThread.join();
long expected = expectedChecksum();
if (checksum == expected) {
System.out.println("ok checksum=" + checksum);
} else {
throw new AssertionError("checksum=" + checksum + ", expected=" + expected);
}
}
}
```
### Observed results
On FEX with `-XX:+UseZGC`, the test failed 3/3:
```text
trial 1: exit 134, fatal SIGSEGV in ZUncoloredRoot::process(...)
trial 2: exit 100, NullPointerException because this.bytes is null
trial 3: exit 100, NullPointerException because this.bytes is null
```
### Controls
The same class passes under FEX with ZGC when compiled code is disabled:
```sh
/path/to/FEX jdk-26.0.2+10/bin/java \
-Xint -Xms128m -Xmx128m -XX:+UseZGC \
FEXZGCBarrierRepro
```
Output:
```text
ok checksum=-4923005376
```
The same class also passes under FEX with the default-style G1 collector:
```sh
/path/to/FEX jdk-26.0.2+10/bin/java \
-Xms64m -Xmx64m -XX:+UseG1GC \
FEXZGCBarrierRepro
```
Output:
```text
ok checksum=-4923005376
```
As a comparator, the same x86-64 JDK and same rootfs pass under QEMU user-mode with ZGC and compiled code enabled:
```sh
qemu-x86_64-static -L /path/to/rootfs \
jdk-26.0.2+10/bin/java \
-Xms128m -Xmx128m -XX:+UseZGC \
FEXZGCBarrierRepro
```
Output:
```text
ok checksum=-4923005376
```
Contributor guide
Research direction
Start with FEXZGCBarrierRepro.java and run the provided compiled-code ZGC command through FEX, then compare the -Xint, G1, and QEMU controls. Done means the compiled-code ZGC run completes without a crash or impossible exception and prints the expected checksum.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp, java
- Domain
- operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100