elastic / elastic/elastic-otel-java
ClassCircularityError: java/lang/invoke/MethodHandle$1 during agent premain on JDK 17 when OTEL_JAVAAGENT_CONFIGURATION_FILE is set (1.8.0 / 1.10.0)
- Dominant language
- Java
- Stars
- 35
- Forks
- 32
- Avg merge
- 1d 8h
- Merged PRs (30d)
- 30
Description
## Summary
EDOT Java **1.8.0** and **1.10.0** crash during agent premain on JDK 17 with `java.lang.ClassCircularityError: java/lang/invoke/MethodHandle$1` when `OTEL_JAVAAGENT_CONFIGURATION_FILE` is set to a properties file containing at least one real `otel.*` entry. **1.7.0**, **1.9.0**, and **1.11.0** are unaffected. An empty file or delivering the same properties via env vars does not trigger.
The bug is rooted in upstream `opentelemetry-java-instrumentation` PR [#15091](https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/15091) and surfaces specifically in distributions (including EDOT) due to additional SPI / autoconfigure class-loading during premain. The bare upstream agent jar does not reproduce.
## Stack trace
```
Picked up JAVA_TOOL_OPTIONS: -javaagent:/javaagent.jar
[main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.8.0
java.lang.ClassCircularityError: java/lang/invoke/MethodHandle$1
at java.base/java.lang.invoke.MethodHandle.customize(MethodHandle.java:1741)
at java.base/java.lang.invoke.MethodHandle.maybeCustomize(MethodHandle.java:1731)
at java.base/java.lang.invoke.Invokers.maybeCustomize(Invokers.java:634)
at java.base/java.lang.invoke.Invokers.checkCustomized(Invokers.java:628)
at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:134)
at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:315)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:281)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:271)
at io.opentelemetry.sdk.autoconfigure.spi.internal.DefaultConfigProperties.(DefaultConfigProperties.java:79)
at io.opentelemetry.sdk.autoconfigure.spi.internal.DefaultConfigProperties.create(DefaultConfigProperties.java:54)
at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.buildImpl(AutoConfiguredOpenTelemetrySdkBuilder.java:452)
...
at co.elastic.otel.agent.ElasticAgent.premain(ElasticAgent.java:39)
```
The visible crash site (`DefaultConfigProperties.:79`) is incidental — the actual race is on the lambda sites in upstream `ConfigurationFile.loadConfigFile()` (see Mechanism).
## Reproducer
Parameterized by EDOT version. Dockerfile/README below
```dockerfile
# syntax=docker/dockerfile:1.4
ARG EDOT_VERSION=1.8.0
FROM docker.elastic.co/observability/elastic-otel-javaagent:${EDOT_VERSION} AS otel-agent
FROM eclipse-temurin:17-jdk
COPY --from=otel-agent /javaagent.jar /javaagent.jar
RUN mkdir -p /opt/otel-repro/extensions \
&& printf 'otel.javaagent.extensions=/opt/otel-repro/extensions\n' > /opt/otel-repro/config.properties
ENV OTEL_JAVAAGENT_CONFIGURATION_FILE=/opt/otel-repro/config.properties
ENV JAVA_TOOL_OPTIONS=-javaagent:/javaagent.jar
CMD ["java", "-version"]
```
```
# Build
docker build --platform=linux/amd64 --build-arg EDOT_VERSION=1.8.0 -t repro:1.8.0 .
# Crash repro — fires the ClassCircularityError above
docker run --platform=linux/amd64 --rm repro:1.8.0
# Workaround — same property delivered via env var, JVM starts cleanly
docker run --platform=linux/amd64 --rm \
-e OTEL_JAVAAGENT_CONFIGURATION_FILE= \
-e OTEL_JAVAAGENT_EXTENSIONS=/opt/otel-repro/extensions \
repro:1.8.0
```
Crash 3/3 deterministic on linux/amd64 (Apple Silicon via Rosetta 2; native arm64 doesn't reliably reproduce due to JIT timing). Cycle `EDOT_VERSION` through `1.7.0 / 1.9.0 / 1.10.0 / 1.11.0` to verify the matrix below.
## Affected versions
| EDOT | Upstream agent | Result |
|------|----------------|--------|
| 1.7.0 | 2.21.0 | clean |
| **1.8.0** | **2.22.0** | crash |
| 1.9.0 | 2.24.0 | clean |
| **1.10.0** | **2.26.1** | crash |
| 1.11.0 | 2.27.0 | clean |
## What rules out other causes
| Hypothesis | Test | Result |
|------------|------|--------|
| Indy advice mode is the trigger | `OTEL_JAVAAGENT_EXPERIMENTAL_INDY=false` on 1.8.0 | still crashes |
| File I/O alone is the trigger | `OTEL_JAVAAGENT_CONFIGURATION_FILE=/dev/null` (empty file) | clean |
| The extensions feature itself is the trigger | Drop file, deliver as `OTEL_JAVAAGENT_EXTENSIONS` env var | clean |
| Multi-agent transformer pressure required | Single javaagent | crashes |
| Application code matters | `java -version`, no app | crashes |
| Bare upstream agent reproduces | Replace EDOT image with v2.22.0 `opentelemetry-javaagent.jar` from upstream releases, identical setup | clean |
The trigger is the configuration-file code path processing real properties during premain on the affected agent versions, in combination with EDOT's SPI loadout.
## Mechanism
`ConfigurationFile.loadConfigFile()` (in upstream, unchanged across all five tags) ends with:
```java
return properties.entrySet().stream()
.collect(toMap(
e -> e.getKey().toString(),
e -> e.getValue().toString()));
```
→ https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/v2.22.0/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/config/ConfigurationFile.java#L80
When the file has content, those two lambda sites need to link via `LambdaMetafactory → BootstrapMethodInvoker → Invokers.checkCustomized → MethodHandle.maybeCustomize → MethodHandle.customize → load MethodHandle$1`. On JDK 17 they race with `MethodHandle$1`'s ongoing definition.
In upstream v2.21.0, `AgentInstaller.start()` called `setupUnsafe(inst)` before reaching the SDK autoconfigure pipeline. That call ran `UnsafeInitializer.initialize()` → `SunMiscUnsafeGenerator.generateUnsafe()`, which generated a replacement `sun.misc.Unsafe` class via classloader machinery and incidentally finished defining `MethodHandle$1` before any later code asked the JDK to link a lambda.
Upstream PR [#15091](https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/15091) "Don't use unsafe on jdk23+" (in 2.22.0) removed `setupUnsafe(inst)` and deleted `UnsafeInitializer` entirely:
```diff
@@ -107,7 +107,6 @@ public static void installBytebuddyAgent(
logVersionInfo();
if (earlyConfig.getBoolean(JAVAAGENT_ENABLED_CONFIG, true)) {
- setupUnsafe(inst);
List agentListeners = loadOrdered(AgentListener.class, extensionClassLoader);
installBytebuddyAgent(inst, extensionClassLoader, agentListeners, earlyConfig);
}
```
After 2.22.0 nothing on the agent's premain path reliably pre-warms `MethodHandle$1`. Whether the lambdas in `loadConfigFile()` link cleanly now depends on what other class-loading happens during premain.
The bare upstream agent doesn't load enough classes during premain to surface the race — `MethodHandle$1` happens to already be defined by the time the lambdas link. EDOT's distribution adds enough SPI / autoconfigure class-loading work (`ElasticAutoConfigurationCustomizerProvider`, `ConfigLoggingAgentListener`, `ElasticDistroResourceProvider`, etc., all auto-discovered via `META-INF/services`) to push the linker into the bad window. That's why the bug is EDOT-visible but bare-upstream-invisible.
The clean upstream releases (2.24.0, 2.27.0) didn't fix the race — `setupUnsafe` did not come back, `UnsafeInitializer.java` is still gone in `main`. Unrelated startup-ordering refactors (signature changes in `installBytebuddyAgent`, `setBootstrapPackages` reordering, #17241 / #16638 / #17824) happen to inject other class loading earlier and incidentally warm the linker. 2.26.0's package-rename refactor shifted ordering back into the bad window. Any future startup refactor can re-surface the race.
## Suggested fix
**EDOT-side (workaround):** add a one-line `MethodHandle$1` preload at the top of `co.elastic.otel.agent.ElasticAgent.premain`, before `OpenTelemetryAgent.premain` is called:
```java
// preload MethodHandle internals to avoid JDK 17 ClassCircularityError on
// the first invokedynamic site that runs during premain (regression in
// upstream agent 2.22.0+ — see )
((java.util.function.Function) (x -> x)).apply(null);
```
This makes EDOT robust regardless of which upstream agent version it pins. Same shape as upstream's existing `ThreadLocalRandom.current()` preload (#14030).
**Upstream-side (better long-term):** sibling preload in `AgentInstaller.installBytebuddyAgent` adjacent to the existing `ThreadLocalRandom.current()` preload at https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/AgentInstaller.java#L114-L120. Fixes this for every distribution, not just EDOT. Open question whether to escalate to upstream after the EDOT-side fix lands.
## Workaround for affected users
Deliver configuration-file properties as env vars or `-D` system properties instead. Replace `otel.javaagent.extensions=/path` in the file with `OTEL_JAVAAGENT_EXTENSIONS=/path` (env var translation: lowercase, replace dots with underscores, uppercase, prepend `OTEL_`). Functionally identical input path to the agent.
`OTEL_JAVAAGENT_EXPERIMENTAL_INDY=false` does **not** prevent this — confirmed empirically.
## Environment
JDK 17.0.18 Temurin, linux/amd64. Reproduced on Apple Silicon via Rosetta 2 (`--platform=linux/amd64`); native linux/arm64 does not reliably reproduce due to JIT timing differences.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.