adoptium / adoptium/adoptium-support

HotSpot-based Java 11 and higher VM crashes when loaded and initialized via JNI Invocation API on AIX

Open
#997 7 comments 0 reactions 0 assignees View on GitHub
bug jbs:reported keep
Dominant language
No language data
Stars
59
Forks
13
PR merge metrics
No merged PRs in 30d

Description

### Please provide a brief summary of the bug

All HotSpot-based distributions that we've tested since 11.0.12 (might have happened earlier) crash very early in the JVM initialization on AIX. The same method that we've been using since around 2003 for loading the JVM still works without fail on our other supported platforms (WIndows and Red Hat-compatible Linux distributions). This includes JDK 21 distributions. As a result of this, we're currently having to embed IBM's OpenJ9-based Semeru JRE in our product distribution, but we recommend the HotSpot-based Adoptium builds to our customers who embed our product in their Java applications (which don't need the JNI Invocation API).

As inferred above, the OpenJ9-based distributions through JDK 21 (highest version that we've tested) load and initialize via JNI Invocation Interface without error on all of our supported platforms, including AIX.

### Did you test with the latest update version?

```Shell
We have tested with the latest available Adoptium JDK 11, 17 and 21 builds for AIX. We've also tested with the latest available SapMachine 21 build for AIX.
```

### Please provide steps to reproduce where possible

[JvmLoader.tar.gz](https://github.com/adoptium/adoptium-support/files/13980256/JvmLoader.tar.gz)

Attached is a tar.gz which contains a small sample program which illustrates the segmentation fault crash in the JVM. To run it, follow these steps:

1. Extract into a folder on an AIX 7.2 machine with sufficient IBM XL C++ runtime
2. Open the file named JvmManager.h, and either add or uncomment one of the lines that initialize a static string variable named JVM_FILE. Replace the value with a path to a JDK 11 or higher version libjvm.so file.
3. Use the included file named "build" to build the small JvmLoader application
4. Run JvmLoader something like the following: LIBPATH=/jdkpath/lib/server:/jdkpath/lib JvmLoader
5. For us, this produces a segmentation fault in all HotSpot JVM versions 11 and higher. This works without fail on all IBM Semeru (which uses OpenJ9 JVM) versions 11 and higher and all Java versions that we've tested (through JDK 21) on Windows and Linux. Note that we didn't include the platform-specific code for Windows and Linux which loads the jvm shared library

### Expected Results

After explicitly loading libjvm.so, the JVM loads successfully when JNI_CreateJavaVM is called.

### Actual Results

After explicitly loading libjvm.so, the JVM crashes with a segmentation fault while calling JNI_CreateJavaVM. The dbx utility reports the following stack trace from our test application:

IPRA.$checked_mprotect__FPcUli(??, ??, ??) at 0x90000003024778c
guard_memory__2osFPcUl(??, ??) at 0x900000030241978
create_stack_guard_pages__10JavaThreadFv(??) at 0x9000000302fd9bc
create_vm__7ThreadsFP14JavaVMInitArgsPb(??, ??) at 0x900000030302470
JNI_CreateJavaVM_inner__FPP7JavaVM_PPvPv(??, ??, ??) at 0x900000030a2f92c
JvmManager::initializeJvm()(), line 2179 in "memory"
JvmLoader.JvmManager::JvmManager()::'lambda'()::operator()() const(this = 0x000000011004a4e0), line 39 in "JvmManager.h"
unnamed block in _ZNSt3__117__call_once_proxyINS_5tupleIJOZN10JvmManagerC1EvEUlvE_EEEEEvPv(__vp = 0x000000011004a4f8), line 2220 in "type_traits"
unnamed block in _ZNSt3__117__call_once_proxyINS_5tupleIJOZN10JvmManagerC1EvEUlvE_EEEEEvPv(__vp = 0x000000011004a4f8), line 2220 in "type_traits"
_ZNSt3__117__call_once_proxyINS_5tupleIJOZN10JvmManagerC1EvEUlvE_EEEEEvPv(__vp = 0x000000011004a4f8), line 2220 in "type_traits"
std::__1::__call_once(unsigned long volatile&, void*, void (*)(void*))(??, ??, ??) at 0x9000000035c37c8
unnamed block in JvmLoader.JvmManager::JvmManager()(this = 0x000000011004a5d0), line 666 in "mutex"
unnamed block in JvmLoader.JvmManager::JvmManager()(this = 0x000000011004a5d0), line 666 in "mutex"
JvmLoader.JvmManager::JvmManager()(this = 0x000000011004a5d0), line 666 in "mutex"
main::$_0::operator()() const(this = 0x0000000110016730), line 7 in "JvmLoader.cpp"
unnamed block in void* std::__1::__thread_proxy >, main::$_0> >(void*)(__vp = 0x0000000110016730), line 2227 in "type_traits"
unnamed block in void* std::__1::__thread_proxy >, main::$_0> >(void*)(__vp = 0x0000000110016730), line 2227 in "type_traits"
void* std::__1::__thread_proxy >, main::$_0> >(void*)(__vp = 0x0000000110016730), line 2227 in "type_traits"

### What Java Version are you using?

openjdk version "11.0.19" 2023-04-18 OpenJDK Runtime Environment Temurin-11.0.19+7 (build 11.0.19+7) OpenJDK 64-Bit Server VM Temurin-11.0.19+7 (build 11.0.19+7, mixed mode)

### What is your operating system and platform?

AIX 7.2 with IBM XL C++ runtime 16.1.0.10 (note that we experience the same crash with many versions of the IBM XL C++ runtime, including Open XL C++ 17.1.x).

### How did you install Java?

Most tests are on JDK/JRE distributions expanded from a tar.gz archive.

### Did it work before?

```Shell
Yes, this approach to loading and initializing the JVM using the JNI Invocation API has worked on all of our supported platforms (Windows, Red Hat-compatible Linux distributions, and AIX) for 20 years. The crash only started happening on AIX after we the version of Java that we embed without application from Java 8 to Java 11. Our other supported platforms continue to work without fail using embedded JRE 11 and higher distributions.
```

### Did you test with other Java versions?

```Shell
openjdk version "11.0.12" 2021-07-20
OpenJDK Runtime Environment Temurin-11.0.12+7 (build 11.0.12+7)
OpenJDK 64-Bit Server VM Temurin-11.0.12+7 (build 11.0.12+7, mixed mode)

openjdk version "11.0.19" 2023-04-18
OpenJDK Runtime Environment Temurin-11.0.19+7 (build 11.0.19+7)
OpenJDK 64-Bit Server VM Temurin-11.0.19+7 (build 11.0.19+7, mixed mode)

openjdk version "17.0.8.1" 2023-08-24
OpenJDK Runtime Environment Temurin-17.0.8.1+1 (build 17.0.8.1+1)
OpenJDK 64-Bit Server VM Temurin-17.0.8.1+1 (build 17.0.8.1+1, mixed mode)

// The following requires Open XL C++ runtime 17.1.x. Included to show that we experience
// the same crash with a JDK built with IBM Open XL C++ 17.1 (and our application also built
// with the same).
openjdk version "21.0.2-ea" 2024-01-16
OpenJDK Runtime Environment SapMachine (build 21.0.2-ea+2)
OpenJDK 64-Bit Server VM SapMachine (build 21.0.2-ea+2, mixed mode)

We've tested with other JDK 11+ HotSpot builds as well. All crash with a segmentation fault on AIX.
```

### Relevant log output

```Shell
No log output. Segmentation fault with core dump only.
```

Contributor guide

Open the contributing guide

Research direction

Start with the attached JvmLoader.tar.gz, JvmManager.h, and the included build script; reproduce on AIX 7.2 using the stated JDK and LIBPATH setup. Trace JNI_CreateJavaVM through create_vm and create_stack_guard_pages using the supplied dbx stack trace. Done means the HotSpot-based libjvm.so loads and JNI_CreateJavaVM completes without a segmentation fault.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, java
Domain
operating-systems
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.