open-telemetry / open-telemetry/opentelemetry-java-instrumentation

Intermittent JVM/runtime metrics missing with OpenTelemetry Java Agent on WebSphere Traditional after server restart

Open
#19,413 16 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
2.6k
Forks
1.2k
Avg merge
2d 16h
Merged PRs (30d)
214

Description

Title:
Intermittent JVM/runtime metrics missing with OpenTelemetry Java Agent on WebSphere Traditional after server restart

Description:
We are integrating OpenTelemetry Java Agent with IBM WebSphere Application Server Traditional on Linux. We have multiple WebSphere JVM instances running on the same node and exporting metrics through a local OpenTelemetry Collector to Prometheus/Grafana.

Issue:
After WebSphere instance restarts, JVM/runtime/application metrics are intermittently missing for 2–3 out of 7 WebSphere instances. The issue is random: after restart, some instances emit metrics successfully while others do not.

Current architecture:
WebSphere Traditional JVMs
-> OpenTelemetry Java Agent
-> Local OpenTelemetry Collector on same Linux node
-> Prometheus exporter
-> Prometheus/Grafana

Collector endpoint:
OTLP HTTP: http://127.0.0.1:4318
Prometheus exporter: http://127.0.0.1:9464/metrics

OTEL Java Agent configuration:
-javaagent:/path/to/opentelemetry-javaagent.jar
-Dotel.metrics.exporter=otlp
-Dotel.traces.exporter=none
-Dotel.logs.exporter=none
-Dotel.exporter.otlp.endpoint=http://127.0.0.1:4318
-Dotel.exporter.otlp.protocol=http/protobuf
-Dotel.metric.export.interval=10000
-Dotel.instrumentation.runtime-metrics.enabled=true

We are also setting unique resource attributes per instance:
-Dotel.service.name=
-Dotel.service.instance.id=--
-Dotel.resource.attributes=host.name=,was.node.name=,was.server.name=,app=,env=

What we observed:

  1. Local OpenTelemetry Collector is running.
  2. Collector debug exporter shows metrics being received when instrumentation works.
  3. Prometheus endpoint works, but metrics for some WebSphere JVM instances are missing intermittently after restart.
  4. IBM Support recommended moving the OTEL -javaagent and -Dotel.* parameters from WebSphere/Candle JVM properties into the WebSphere Linux startup script to ensure consistent agent loading.
  5. We are validating whether missing metrics are caused by the Java agent not being loaded consistently, or by metric export/collection behavior after restart.

Questions:

  1. Is WebSphere Traditional expected to work reliably with OpenTelemetry Java Agent metrics using OTLP HTTP?
  2. Are there known issues with WebSphere Traditional, IBM JDK 8, and OpenTelemetry Java Agent metrics after restart?
  3. Is startup-script injection preferred over WebSphere admin console/custom JVM properties for -javaagent?
  4. Are there recommended JVM flags for multiple WebSphere Traditional JVMs on the same Linux node?
  5. What logs/debug flags should we enable to confirm whether the agent is loaded and metrics exporter is active?

Environment:

  • WebSphere Application Server Traditional version:
  • WAS fix pack:
  • IBM JDK version:
  • Linux OS version:
  • OpenTelemetry Java Agent version:
  • OpenTelemetry Collector version:
  • Number of WebSphere JVMs on node: 7
  • Export protocol: OTLP HTTP
  • Metrics backend: Prometheus/Grafana

Validation commands used:
ps -ef | grep java | grep opentelemetry
grep -i "opentelemetry-javaagent" SystemOut.log
grep -i "opentelemetry-javaagent" native_stderr.log
curl http://127.0.0.1:9464/metrics

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the WebSphere startup configuration, the OpenTelemetry Java Agent settings, and the listed SystemOut.log and native_stderr.log checks. Compare instances that emit metrics with those that do not after restart, using the process and log commands plus the Collector debug exporter and Prometheus endpoint. Done means identifying whether agent loading or metric export/collection accounts for the missing instances, with the relevant version details filled in.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.