elastic / elastic/elastic-otel-java

ClassCircularityError: java/lang/invoke/MethodHandle$1 during agent premain on JDK 17 when OTEL_JAVAAGENT_CONFIGURATION_FILE is set (1.8.0 / 1.10.0)

Open
#1,092 11 comments 0 reactions 0 assignees View on GitHub
agent-java
Dominant language
Java
Stars
35
Forks
32
Avg merge
1d 8h
Merged PRs (30d)
30

Description

## Summary

EDOT Java **1.8.0** and **1.10.0** crash during agent premain on JDK 17 with `java.lang.ClassCircularityError: java/lang/invoke/MethodHandle$1` when `OTEL_JAVAAGENT_CONFIGURATION_FILE` is set to a properties file containing at least one real `otel.*` entry. **1.7.0**, **1.9.0**, and **1.11.0** are unaffected. An empty file or delivering the same properties via env vars does not trigger.

The bug is rooted in upstream `opentelemetry-java-instrumentation` PR [#15091](https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/15091) and surfaces specifically in distributions (including EDOT) due to additional SPI / autoconfigure class-loading during premain. The bare upstream agent jar does not reproduce.

## Stack trace

```
Picked up JAVA_TOOL_OPTIONS: -javaagent:/javaagent.jar
[main] INFO io.opentelemetry.javaagent.tooling.VersionLogger - opentelemetry-javaagent - version: 1.8.0
java.lang.ClassCircularityError: java/lang/invoke/MethodHandle$1
at java.base/java.lang.invoke.MethodHandle.customize(MethodHandle.java:1741)
at java.base/java.lang.invoke.MethodHandle.maybeCustomize(MethodHandle.java:1731)
at java.base/java.lang.invoke.Invokers.maybeCustomize(Invokers.java:634)
at java.base/java.lang.invoke.Invokers.checkCustomized(Invokers.java:628)
at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:134)
at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:315)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:281)
at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:271)
at io.opentelemetry.sdk.autoconfigure.spi.internal.DefaultConfigProperties.(DefaultConfigProperties.java:79)
at io.opentelemetry.sdk.autoconfigure.spi.internal.DefaultConfigProperties.create(DefaultConfigProperties.java:54)
at io.opentelemetry.sdk.autoconfigure.AutoConfiguredOpenTelemetrySdkBuilder.buildImpl(AutoConfiguredOpenTelemetrySdkBuilder.java:452)
...
at co.elastic.otel.agent.ElasticAgent.premain(ElasticAgent.java:39)
```

The visible crash site (`DefaultConfigProperties.:79`) is incidental — the actual race is on the lambda sites in upstream `ConfigurationFile.loadConfigFile()` (see Mechanism).

## Reproducer

Parameterized by EDOT version. Dockerfile/README below

```dockerfile
# syntax=docker/dockerfile:1.4
ARG EDOT_VERSION=1.8.0
FROM docker.elastic.co/observability/elastic-otel-javaagent:${EDOT_VERSION} AS otel-agent
FROM eclipse-temurin:17-jdk
COPY --from=otel-agent /javaagent.jar /javaagent.jar
RUN mkdir -p /opt/otel-repro/extensions \
&& printf 'otel.javaagent.extensions=/opt/otel-repro/extensions\n' > /opt/otel-repro/config.properties
ENV OTEL_JAVAAGENT_CONFIGURATION_FILE=/opt/otel-repro/config.properties
ENV JAVA_TOOL_OPTIONS=-javaagent:/javaagent.jar
CMD ["java", "-version"]
```

```
# Build
docker build --platform=linux/amd64 --build-arg EDOT_VERSION=1.8.0 -t repro:1.8.0 .

# Crash repro — fires the ClassCircularityError above
docker run --platform=linux/amd64 --rm repro:1.8.0

# Workaround — same property delivered via env var, JVM starts cleanly
docker run --platform=linux/amd64 --rm \
-e OTEL_JAVAAGENT_CONFIGURATION_FILE= \
-e OTEL_JAVAAGENT_EXTENSIONS=/opt/otel-repro/extensions \
repro:1.8.0
```

Crash 3/3 deterministic on linux/amd64 (Apple Silicon via Rosetta 2; native arm64 doesn't reliably reproduce due to JIT timing). Cycle `EDOT_VERSION` through `1.7.0 / 1.9.0 / 1.10.0 / 1.11.0` to verify the matrix below.

## Affected versions

| EDOT | Upstream agent | Result |
|------|----------------|--------|
| 1.7.0 | 2.21.0 | clean |
| **1.8.0** | **2.22.0** | crash |
| 1.9.0 | 2.24.0 | clean |
| **1.10.0** | **2.26.1** | crash |
| 1.11.0 | 2.27.0 | clean |

## What rules out other causes

| Hypothesis | Test | Result |
|------------|------|--------|
| Indy advice mode is the trigger | `OTEL_JAVAAGENT_EXPERIMENTAL_INDY=false` on 1.8.0 | still crashes |
| File I/O alone is the trigger | `OTEL_JAVAAGENT_CONFIGURATION_FILE=/dev/null` (empty file) | clean |
| The extensions feature itself is the trigger | Drop file, deliver as `OTEL_JAVAAGENT_EXTENSIONS` env var | clean |
| Multi-agent transformer pressure required | Single javaagent | crashes |
| Application code matters | `java -version`, no app | crashes |
| Bare upstream agent reproduces | Replace EDOT image with v2.22.0 `opentelemetry-javaagent.jar` from upstream releases, identical setup | clean |

The trigger is the configuration-file code path processing real properties during premain on the affected agent versions, in combination with EDOT's SPI loadout.

## Mechanism

`ConfigurationFile.loadConfigFile()` (in upstream, unchanged across all five tags) ends with:

```java
return properties.entrySet().stream()
.collect(toMap(
e -> e.getKey().toString(),
e -> e.getValue().toString()));
```

→ https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/v2.22.0/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/config/ConfigurationFile.java#L80

When the file has content, those two lambda sites need to link via `LambdaMetafactory → BootstrapMethodInvoker → Invokers.checkCustomized → MethodHandle.maybeCustomize → MethodHandle.customize → load MethodHandle$1`. On JDK 17 they race with `MethodHandle$1`'s ongoing definition.

In upstream v2.21.0, `AgentInstaller.start()` called `setupUnsafe(inst)` before reaching the SDK autoconfigure pipeline. That call ran `UnsafeInitializer.initialize()` → `SunMiscUnsafeGenerator.generateUnsafe()`, which generated a replacement `sun.misc.Unsafe` class via classloader machinery and incidentally finished defining `MethodHandle$1` before any later code asked the JDK to link a lambda.

Upstream PR [#15091](https://github.com/open-telemetry/opentelemetry-java-instrumentation/pull/15091) "Don't use unsafe on jdk23+" (in 2.22.0) removed `setupUnsafe(inst)` and deleted `UnsafeInitializer` entirely:

```diff
@@ -107,7 +107,6 @@ public static void installBytebuddyAgent(
logVersionInfo();
if (earlyConfig.getBoolean(JAVAAGENT_ENABLED_CONFIG, true)) {
- setupUnsafe(inst);
List agentListeners = loadOrdered(AgentListener.class, extensionClassLoader);
installBytebuddyAgent(inst, extensionClassLoader, agentListeners, earlyConfig);
}
```

After 2.22.0 nothing on the agent's premain path reliably pre-warms `MethodHandle$1`. Whether the lambdas in `loadConfigFile()` link cleanly now depends on what other class-loading happens during premain.

The bare upstream agent doesn't load enough classes during premain to surface the race — `MethodHandle$1` happens to already be defined by the time the lambdas link. EDOT's distribution adds enough SPI / autoconfigure class-loading work (`ElasticAutoConfigurationCustomizerProvider`, `ConfigLoggingAgentListener`, `ElasticDistroResourceProvider`, etc., all auto-discovered via `META-INF/services`) to push the linker into the bad window. That's why the bug is EDOT-visible but bare-upstream-invisible.

The clean upstream releases (2.24.0, 2.27.0) didn't fix the race — `setupUnsafe` did not come back, `UnsafeInitializer.java` is still gone in `main`. Unrelated startup-ordering refactors (signature changes in `installBytebuddyAgent`, `setBootstrapPackages` reordering, #17241 / #16638 / #17824) happen to inject other class loading earlier and incidentally warm the linker. 2.26.0's package-rename refactor shifted ordering back into the bad window. Any future startup refactor can re-surface the race.

## Suggested fix

**EDOT-side (workaround):** add a one-line `MethodHandle$1` preload at the top of `co.elastic.otel.agent.ElasticAgent.premain`, before `OpenTelemetryAgent.premain` is called:

```java
// preload MethodHandle internals to avoid JDK 17 ClassCircularityError on
// the first invokedynamic site that runs during premain (regression in
// upstream agent 2.22.0+ — see )
((java.util.function.Function) (x -> x)).apply(null);
```

This makes EDOT robust regardless of which upstream agent version it pins. Same shape as upstream's existing `ThreadLocalRandom.current()` preload (#14030).

**Upstream-side (better long-term):** sibling preload in `AgentInstaller.installBytebuddyAgent` adjacent to the existing `ThreadLocalRandom.current()` preload at https://github.com/open-telemetry/opentelemetry-java-instrumentation/blob/main/javaagent-tooling/src/main/java/io/opentelemetry/javaagent/tooling/AgentInstaller.java#L114-L120. Fixes this for every distribution, not just EDOT. Open question whether to escalate to upstream after the EDOT-side fix lands.

## Workaround for affected users

Deliver configuration-file properties as env vars or `-D` system properties instead. Replace `otel.javaagent.extensions=/path` in the file with `OTEL_JAVAAGENT_EXTENSIONS=/path` (env var translation: lowercase, replace dots with underscores, uppercase, prepend `OTEL_`). Functionally identical input path to the agent.

`OTEL_JAVAAGENT_EXPERIMENTAL_INDY=false` does **not** prevent this — confirmed empirically.

## Environment

JDK 17.0.18 Temurin, linux/amd64. Reproduced on Apple Silicon via Rosetta 2 (`--platform=linux/amd64`); native linux/arm64 does not reliably reproduce due to JIT timing differences.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.