apache / apache/maven-surefire

3.6.0 regression: forked JVM silently vanishes on Windows for tests using FFM upcalls from background threads

Open
#3,459 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
461
Forks
588
Avg merge
1d 8h
Merged PRs (30d)
19

Description

## Affected version
3.6.0 (regression). Bisected to commit
[90529e9](https://github.com/apache/maven-surefire/commit/90529e9fab1a2df720d9a88c1abb6c34a4ab5336)
("Use StackWalker in StackTraceProvider when available (Java 9+)", #3374).
Last known good: 3.6.0-M1 and everything before that commit.

## Environment
- OS: windows-latest GitHub Actions runner (Windows Server 2022/2025)
- JDK: Zulu 25.0.4-7, x64
- Build: Maven, `reuseForks=false`, `argLine` includes `--enable-native-access=ALL-UNNAMED`
- Project: https://github.com/dfa1/rocksdb-ffm (Java FFM bindings for RocksDB's C API)

## Bug description

Bumping `maven-surefire-plugin` from 3.5.6 to 3.6.0 makes three specific test
classes deterministically kill their forked JVM on `windows-latest` only.
macOS and Linux (x86_64 and aarch64) are unaffected. The same commit set with
surefire pinned at 3.5.6 (or at 3.6.0-M1) passes on Windows every time.

```
[ERROR] ExecutionException The forked VM terminated without properly saying goodbye. VM crash or System.exit called?
[ERROR] Crashed tests:
io.github.dfa1.rocksdbffm.CompactionFilterFactoryTest
io.github.dfa1.rocksdbffm.CompactionFilterTest
io.github.dfa1.rocksdbffm.EventNotifierTest
```

This reproduced identically across many independent CI runs — always exactly
these three classes, never a different subset.

### What we ruled out

- **Not a classic native crash**: no `hs_err_pid*.log` is ever produced,
despite surefire's own message suggesting a VM crash. If HotSpot's signal
handler had caught a fault normally, that file would exist.
- **Not the "Boot Manifest-JAR" classpath warning**: every single fork
(150+, since `reuseForks=false`) logs
`Boot Manifest-JAR contains absolute paths in classpath '...', 'other' has
different root` to the `.dumpstream` file (a `Path.relativize()` failure
across Windows drive roots, falling back to absolute paths in the manifest
jar). This happens for both passing and crashing forks alike — harmless
noise, unrelated to the real bug.
- **Not simply "any test with a native upcall crashes"**: other tests in the
same suite that register FFM upcalls but only ever invoke them
synchronously on the JVM's own calling thread (e.g. a custom merge
operator callback, or a test that calls `System.exit` from a JVM-owned
thread) pass without issue.

### Bisection

We built surefire from source at each candidate commit between
`surefire-3.6.0-M1` and `surefire-3.6.0` (54 commits) and ran our suite
against each, narrowing by binary search:

| Round | Commit (index in the 54) | Result |
|---|---|---|
| baseline | 3.6.0-M1 | pass |
| 1 | `37f0725` (26) | crash |
| 2 | `381c404` (12) | crash |
| 3 | `c633042` (5) | pass |
| 4 | `90529e9` (8) | **crash** |
| 5 | `e759c64` (6) | pass |
| 6 (final) | `523e1ce` (7) | pass |

Index 7 passing and index 8 crashing pins the boundary exactly: commit
`90529e9` is the first bad commit in the range.

The regression is isolated to that single commit,
[90529e9](https://github.com/apache/maven-surefire/commit/90529e9fab1a2df720d9a88c1abb6c34a4ab5336)
/ PR [#3374](https://github.com/apache/maven-surefire/pull/3374), which
switches `StackTraceProvider.getStack()` from `Thread.currentThread().getStackTrace()`
to `java.lang.StackWalker.getInstance(options, depth).walk(...)` (falling back
to the old path only if the `StackWalker` call throws).

### Suspected mechanism

`CompactionFilter`/`CompactionFilterFactory`/`EventNotifier` register a
native → Java callback that RocksDB invokes from its own background threads
(the compaction thread pool, and flush/compaction event-listener threads) —
not threads the JVM created. The callback path is a JDK 25
`Linker.upcallStub()` entry point, which the JVM implements as a
generated hidden class/`MethodHandle` trampoline. The first time such a
background thread fires the upcall, the JVM attaches that foreign thread,
and from the JVM's point of view its call stack starts at that synthetic
hidden-class frame rather than a normal `Thread.run()`-rooted chain.

`Thread.getStackTrace()` (still used as the fallback in the patched code)
tolerates that stack shape fine — this project's whole suite passed under
every surefire version before this commit. `StackWalker`, by contrast,
does extra classification work on each frame specifically to hide
reflection/lambda frames (per the PR's own code comment), which means it's
doing more invasive introspection of frame metadata for exactly the kind of
synthetic/hidden frame an FFM upcall trampoline produces. We suspect that
introspection is hitting a rough edge in HotSpot's stack-walking internals
for hidden-class/MethodHandle frames belonging to a JVM-attached (not
JVM-created) thread, causing a hard, in-process JVM fault severe enough that
the normal crash-dump path (`hs_err_pid*.log`) never runs.

## Reproduction

Not yet minimized to a surefire-only reproducer (would require a small
native library exercising a JDK 25 `Linker.upcallStub` invoked from a
non-JVM-created thread, run under `StackTraceProvider.getStack()`'s new
path). Happy to attempt one if useful — our current reproduction is the
full `dfa1/rocksdb-ffm` CI matrix.

Happy to run additional diagnostics on our CI if that helps narrow this
down further.

Contributor guide

Open the contributing guide

Research direction

Reproduce the failure in the dfa1/rocksdb-ffm suite on a Windows GitHub Actions runner with JDK 25, focusing on CompactionFilterFactoryTest, CompactionFilterTest, and EventNotifierTest. Read StackTraceProvider.getStack() and commit 90529e9, then compare the StackWalker and fallback paths; done means the affected forked tests complete without the JVM silently terminating.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, java
Domain
build-system, testing-qa
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.