open-telemetry / open-telemetry/opentelemetry-java-instrumentation

K8s OpenTelemetry Java Agent, missing metrics?

Open
#18,832 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Java
Stars
2.6k
Forks
1.2k
Avg merge
2d 18h
Merged PRs (30d)
228

Description

Hi,

We are migrating our Wildfly VM's to Kubernetes. We used to use Prometheus to scrape metrics but we want to switch to OpenTelemetry. We are currently using the OpenTelemetry Operator with the Instrumentation CR:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: java-instrumentation
  namespace: monitoring
spec:
  exporter:
    endpoint: http://alloy-receiver.kube-monitoring:4318
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: "1"
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:2.27.0
    env:
      - name: OTEL_EXPORTER_OTLP_PROTOCOL
        value: http/protobuf
      - name: OTEL_JMX_TARGET_SYSTEM
        value: wildfly
      - name: OTEL_METRICS_EXPORTER
        value: otlp

This works, but we are missing some metrics that we hope to be able to get. For example, with Prometheus we get wildfly_undertow_active_sessions. Now with OTEL we getwildfly_active_sessions but this value stays empty. There should also be metrics for JDBC connections, especially one for max pool connections, but we do not seem to get them via OTEL. We are not sure what we are missing. If I understand correctly, by setting the OTEL_JMX_TARGET_SYSTEM to wildfly, we should already receive a lot of these metrics. I read about the JDBC metrics here (https://github.com/open-telemetry/semantic-conventions/blob/main/docs/db/database-metrics.md), but I am not sure if these include the max pool connections metric.

What do we need to set to get the JDBC (and undertow if it is indeed not the same) metric?

Sorry about the vague issue, I'm just not sure on how to continue. I also wasn't sure if I should set this here or in the opentelemetry-operator repo.

Thanks a lot!

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Begin with the Java agent's JMX instrumentation path and the OTEL_JMX_TARGET_SYSTEM=wildfly configuration; compare emitted wildfly_active_sessions with the expected Undertow and JDBC metrics. Reproduce using the supplied Instrumentation CR, then determine whether additional configuration is required or the metrics are unsupported. Done means the required settings or a concrete instrumentation gap is identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, kubernetes
Domain
backend, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.