JRE (Likely JVM) fails (SIGSEGV) on x86 Ubuntu 24.04 LTS emulated on Apple Silicon M2 ARM
- Dominant language
- Swift
- Stars
- 35.5k
- Forks
- 1.8k
- Avg merge
- 5d 5h
- Merged PRs (30d)
- 7
Description
# Issue
JRE (HotSpot Runtime) errors with SIGSEGV on x86 Linux Ubuntu 24.04.2 LTS when it is emulated on Apple Silicon M2. In this case, JRE is being triggered by SBT that is running Scala source code.
This could be a Qemu issue, an OpenJDK issue, an Apple issue, etc. - Let me know if this is the wrong place/not under the purview of UTM and I'll post it somewhere else.
## Error
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x000074e2e98b2f00, pid=1191, tid=1192
#
# JRE version: OpenJDK Runtime Environment (20.0.2) (build 20.0.2-internal-adhoc..src)
# Java VM: OpenJDK 64-Bit Server VM (20.0.2-internal-adhoc..src, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# J 8447 c1 sbt.internal.util.Util$.prependEither(Lscala/Tuple2;Lscala/util/Either;)Lscala/Tuple2; (135 bytes) @ 0x000074e2e98b2f00 [0x000074e2e98b2ee0+0x0000000000000020]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /scratch/ofo/ofot-cy-vm/core.1191)
#
# An error report file with more information is saved as:
# /scratch/ofo/ofot-cy-vm/hs_err_pid1191.log
[41.212s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
```
The error seems to be non-deterministic -- Running the same command to trigger the JRE, with the same settings, again results in the following error -- note the different `Problematic frame`:
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007852e9235f40, pid=1382, tid=1383
#
# JRE version: OpenJDK Runtime Environment (20.0.2) (build 20.0.2-internal-adhoc..src)
# Java VM: OpenJDK 64-Bit Server VM (20.0.2-internal-adhoc..src, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# J 4536 c1 sbt.internal.util.EvaluateSettings$$anon$1.apply(Ljava/lang/Object;)Ljava/lang/Object; (9 bytes) @ 0x00007852e9235f40 [0x00007852e9235f20+0x0000000000000020]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /scratch/ofo/ofot-cy-vm/core.1382)
#
# An error report file with more information is saved as:
# /scratch/ofo/ofot-cy-vm/hs_err_pid1382.log
[93.721s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
make: *** [/scratch/ofo/ofot-cy-vm/common.mk:409: launch-sbt] Aborted (core dumped)
```
## Background information
I am attempting to run a Scala project (https://github.com/ucb-bar/chipyard) on a x86 machine emulated on an Apple Silicon device. The project build flow fails on step 5 when Scala sources are compiled and run. You can reproduce the issue by running Chipyard's recommended setup flow here:
https://chipyard.readthedocs.io/en/stable/Chipyard-Basics/Initial-Repo-Setup.html#default-requirements-installation
Then instead of running the given `build-setup` command in the tutorial, run `./build-setup.sh riscv-tools -s 3 -s 8 -s 7 -s 8 -s 9 -s 10 --use-lean-conda` in order to skip the irrevelent setup steps.
The SBT build config is in the project's base directory under `build.sbt`. There is a `commonSettings` sequence that is inherited by each subsequent project. The flow: line 409 of [common.mk](https://github.com/ucb-bar/chipyard/blob/main/common.mk) is triggered by line 257 & 258 of [build-setup.sh](https://github.com/ucb-bar/chipyard/blob/main/scripts/build-setup.sh), which then triggers SBT with some arguments passed into the SBT executable.
## Initial Research & Attempted Fixes
Came across https://github.com/docker/for-mac/issues/7006 and https://github.com/docker/for-mac/issues/7286 which seem to be similar to the issue here. I've attempted the following fixes based on the 2 issues above:
1. Force JVM garbage collector flags in `build.sbt`
In `build.sbt`, under `commonSettings` sequence, add the following options:
```
fork := true,
javaOptions ++= Seq(
"-XX:-UseG1GC",
"-XX:+UseParallelGC"
),
```
I've also tried: `-XX:+UseSerialGC`, `-XX:+UseZGC` + `-XX:+UnlockExperimentalVMOptions` in place of `-XX:+UseParallelGC`.
In addition, I've tried the following syntax changes `-XX:+UseSerialGC` => `J-XX:+UseSerialGC` and `+UseSerialGC`. These cases either resulted in SIGSEGV or SIGILL.
I've also tried removing `fork := true`, which makes no difference
A sample SIGILL error looks like the following:
```
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x000077a595162b24, pid=2371, tid=2372
#
# JRE version: OpenJDK Runtime Environment (20.0.2) (build 20.0.2-internal-adhoc..src)
# Java VM: OpenJDK 64-Bit Server VM (20.0.2-internal-adhoc..src, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# J 3883 c1 sbt.Scope$.resolveReference(Ljava/net/URI;Lscala/Function1;Lsbt/Reference;)Lsbt/ResolvedReference; (66 bytes) @ 0x000077a595162b24 [0x000077a595162b00+0x0000000000000024]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /scratch/ofo/ofot-cy-vm/core.2371)
#
# An error report file with more information is saved as:
# /scratch/ofo/ofot-cy-vm/hs_err_pid2371.log
[39.397s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
make: *** [/scratch/ofo/ofot-cy-vm/common.mk:409: launch-sbt] Aborted (core dumped)
```
2. Force garbage collector flags via environment variables
Setting the environment variable `JAVA_TOOL_OPTIONS="-XX:-UseG1GC -XX:+UseSerialGC"`
This results in the following SIGILL:
```
========== BEGINNING STEP 5: Pre-compiling Chipyard Scala sources ==========
/scratch/ofo/ofot-cy-vm/sims/verilator /scratch/ofo/ofot-cy-vm
Running with RISCV=/scratch/ofo/ofot-cy-vm/.conda-env/riscv-tools
cd /scratch/ofo/ofot-cy-vm && java -jar /scratch/ofo/ofot-cy-vm/scripts/sbt-launch.jar -Dsbt.ivy.home=/scratch/ofo/ofot-cy-vm/.ivy2 -Dsbt.global.base=/scratch/ofo/ofot-cy-vm/.sbt -Dsbt.boot.directory=/scratch/ofo/ofot-cy-vm/.sbt/boot/ -Dsbt.color=always -Dsbt.supershell=false -Dsbt.server.forcestart=true ";project chipyard; compile"
Picked up JAVA_TOOL_OPTIONS: -XX:-UseG1GC -XX:+UseSerialGC
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x00007bec84c23b24, pid=9621, tid=9622
#
# JRE version: OpenJDK Runtime Environment (20.0.2) (build 20.0.2-internal-adhoc..src)
# Java VM: OpenJDK 64-Bit Server VM (20.0.2-internal-adhoc..src, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, serial gc, linux-amd64)
# Problematic frame:
# J 112 c1 java.lang.ref.Reference.reachabilityFence(Ljava/lang/Object;)V java.base@20.0.2-internal (1 bytes) @ 0x00007bec84c23b24 [0x00007bec84c23b20+0x0000000000000004]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /scratch/ofo/ofot-cy-vm/core.9621)
#
# An error report file with more information is saved as:
# /scratch/ofo/ofot-cy-vm/hs_err_pid9621.log
[1.775s][warning][os] Loading hsdis library failed
#
# If you would like to submit a bug report, please visit:
# https://bugreport.java.com/bugreport/crash.jsp
#
make: *** [/scratch/ofo/ofot-cy-vm/common.mk:409: launch-sbt] Aborted (core dumped)
build-setup.sh: Build script failed with exit code 2 at step 5: Pre-compiling Chipyard Scala sources
```
3. Set `JAVA_OPTS="-XX:-UseG1GC -XX:+UseSerialGC"`, along with putting these in a `.jvmopts` file in the root project directory -- results in following SIGILL
```
========== BEGINNING STEP 5: Pre-compiling Chipyard Scala sources ==========
/scratch/ofo/ofot-cy-vm/sims/verilator /scratch/ofo/ofot-cy-vm
Running with RISCV=/scratch/ofo/ofot-cy-vm/.conda-env/riscv-tools
cd /scratch/ofo/ofot-cy-vm && java -jar /scratch/ofo/ofot-cy-vm/scripts/sbt-launch.jar -Dsbt.ivy.home=/scratch/ofo/ofot-cy-vm/.ivy2 -Dsbt.global.base=/scratch/ofo/ofot-cy-vm/.sbt -Dsbt.boot.directory=/scratch/ofo/ofot-cy-vm/.sbt/boot/ -Dsbt.color=always -Dsbt.supershell=false -Dsbt.server.forcestart=true ";project chipyard; compile"
Picked up JAVA_TOOL_OPTIONS: -Xmx8G -Xss8M -Djava.io.tmpdir=/scratch/ofo/ofot-cy-vm/.java_tmp
[info] welcome to sbt 1.8.2 (N/A Java 20.0.2-internal)
[info] loading settings for project ofot-cy-vm-build from plugins.sbt ...
[info] loading project definition from /scratch/ofo/ofot-cy-vm/project
#
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGILL (0x4) at pc=0x000072ca594323a5, pid=10756, tid=10811
#
# JRE version: OpenJDK Runtime Environment (20.0.2) (build 20.0.2-internal-adhoc..src)
# Java VM: OpenJDK 64-Bit Server VM (20.0.2-internal-adhoc..src, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# J 9107 c1 sjsonnew.shaded.org.typelevel.jawn.StringParser.at(I)C (9 bytes) @ 0x000072ca594323a5 [0x000072ca59432380+0x0000000000000025]
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport -p%p -s%s -c%c -d%d -P%P -u%u -g%g -- %E" (or dumping to /scratch/ofo/ofot-cy-vm/core.10756)
#
# An error report file with more information is saved as:
# /scratch/ofo/ofot-cy-vm/hs_err_pid10756.log
```
4. Add garbage collector flag directly to the SBT command in `common.mk` line 409
Change `cd $(base_dir) && $(SBT) "$(SBT_COMMAND)"` to `cd $(base_dir) && $(SBT) -XX:-UseG1GC -XX:+UseSerialGC "$(SBT_COMMAND)"`
Also adding this directly to the `make launch-sbt SBT_COMMAND=";project chipyard; compile"` in line 257 of build-setup.sh
Results in:
```
# A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x00007c59991033c0, pid=8242, tid=8243
#
# JRE version: OpenJDK Runtime Environment (20.0.2) (build 20.0.2-internal-adhoc..src)
# Java VM: OpenJDK 64-Bit Server VM (20.0.2-internal-adhoc..src, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Problematic frame:
# J 3661 c1 sbt.Project$$$Lambda$368+0x00007c5930343438.apply(Ljava/lang/Object;)Ljava/lang/Object; (12 bytes) @ 0x00007c59991033c0 [0x00007c59991033a0+0x0000000000000020]
```
5. Single Core Machine
Reducing the UTM machine to 1 core seem to allow compilation & execution of the Scala sources without issue (although excruciatingly slow) -- but it works. Not sure why. Initially my guess was with a problem with how mostly concurrent garbage collectors are handled in emulation by Qemu and MacOS, but this doesn't explain why forcing the Serial GC still results in a Segfault.
At this point, I am starting to doubt it is a garbage collector issue like it was in the Docker issue.
# Configuration
* UTM Version: 4.6.4 (107)
* macOS Version: Sequoia 15.4 Beta (24E5222f)
* Mac Chip (Intel, M1, ...): M2 Max, 32GB Memory, A2779
* Emulated System: `uname -a: Linux ofo 6.8.0-55-generic #57-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb 12 23:42:21 UTC 2025 x86_64 GNU/Linux` - Q35 + ICH9, 2009 machine, 4 CPU Intel Skylake, 8GB Memory
Qemu Command
qemu-system-x86_64 -L /Users/jimfang/Library/Containers/com.utmapp.UTM/Data/Library/Caches/qemu -S -spice unix=on,addr=0E6854CD-4336-450B-BD55-6E98C26F5D96.spice,disable-ticketing=on,image-compression=off,playback-compression=off,streaming-video=off,gl=off -chardev spiceport,name=org.qemu.monitor.qmp.0,id=org.qemu.monitor.qmp -mon chardev=org.qemu.monitor.qmp,mode=control -nodefaults -vga none -device e1000,mac=A2:B5:C2:87:87:E1,netdev=net0 -netdev vmnet-bridged,id=net0,ifname=en0,isolated=on -device virtio-gpu-pci -cpu Skylake-Client -smp cpus=4,sockets=1,cores=4,threads=1 -machine q35,vmport=off,i8042=off,hpet=off -accel tcg,thread=multi,tb-size=2048 -global PIIX4_PM.disable_s3=1 -global ICH9-LPC.disable_s3=1 -drive if=pflash,format=raw,unit=0,file.filename=/Users/jimfang/Library/Containers/com.utmapp.UTM/Data/Library/Caches/qemu/edk2-x86_64-code.fd,file.locking=off,readonly=on -drive if=pflash,unit=1,file=/Users/jimfang/Library/Containers/com.utmapp.UTM/Data/Documents/Linux.utm/Data/efi_vars.fd -m 8192 -audiodev spice,id=audio0 -device intel-hda -device hda-duplex,audiodev=audio0 -usb -device usb-tablet,bus=usb-bus.0 -device usb-mouse,bus=usb-bus.0 -device usb-kbd,bus=usb-bus.0 -device nec-usb-xhci,id=usb-controller-0 -chardev spicevmc,name=usbredir,id=usbredirchardev0 -device usb-redir,chardev=usbredirchardev0,id=usbredirdev0,bus=usb-controller-0.0 -chardev spicevmc,name=usbredir,id=usbredirchardev1 -device usb-redir,chardev=usbredirchardev1,id=usbredirdev1,bus=usb-controller-0.0 -chardev spicevmc,name=usbredir,id=usbredirchardev2 -device usb-redir,chardev=usbredirchardev2,id=usbredirdev2,bus=usb-controller-0.0 -device ide-cd,bus=ide.0,drive=driveD91FA485-7AB7-4665-A5F9-7ADCA5814EB3,bootindex=0 -drive if=none,media=cdrom,id=driveD91FA485-7AB7-4665-A5F9-7ADCA5814EB3,readonly=on -device ide-hd,bus=ide.1,drive=drive84BC6CF4-BCB2-462C-BBBC-7022C080A4EF,bootindex=1 -drive if=none,media=disk,id=drive84BC6CF4-BCB2-462C-BBBC-7022C080A4EF,file.filename=/Users/jimfang/Library/Containers/com.utmapp.UTM/Data/Documents/Linux.utm/Data/84BC6CF4-BCB2-462C-BBBC-7022C080A4EF.qcow2,discard=unmap,detect-zeroes=unmap -device virtio-serial -device virtserialport,chardev=org.qemu.guest_agent,name=org.qemu.guest_agent.0 -chardev spiceport,name=org.qemu.guest_agent.0,id=org.qemu.guest_agent -device virtserialport,chardev=vdagent,name=com.redhat.spice.0 -chardev spicevmc,id=vdagent,debug=0,name=vdagent -name Linux -uuid 0E6854CD-4336-450B-BD55-6E98C26F5D96 -device virtio-rng-pci -netdev vmnet-shared,id=net1 -device e1000,netdev=net1
# Logs
## UTM Debug Log
[debug.log](https://github.com/user-attachments/files/19412801/debug.log)
## Corresponding JRE Log (No JVM garbage collector flags)
[hs_err_pid1124.log](https://github.com/user-attachments/files/19412802/hs_err_pid1124.log)
More logs for different scenarios can be provided upon request!
Contributor guide
Assessment
This issue has not been assessed yet.