open-telemetry / open-telemetry/opentelemetry-java-instrumentation

HTTP Instrumentation on Quarkus/Netty doesn't work for TLS + HTTP/2

Open
#8,927 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
2.6k
Forks
1.2k
Avg merge
2d 18h
Merged PRs (30d)
228

Description

Describe the bug

When I start Keycloak 21.1.2 or 22.0.0 (which is based on Quarkus and Netty) with TLS and HTTP/2, I don't get http_server_duration_milliseconds* metrics.

When I start it with HTTP only (which will fallback to HTTP/1.1), I get them:

http_server_duration_milliseconds_count{otel_scope_name="io.opentelemetry.netty-4.1",otel_scope_version="1.27.0-alpha",http_method="GET",http_route="/",http_scheme="http",http_status_code="200",net_host_name="localhost",net_host_port="8080",net_protocol_name="http",net_protocol_version="1.1"} 1.0 1689095194824

Steps to reproduce

  • Download Keycloak 22.0.0 and extract - https://www.keycloak.org/archive/downloads-22.0.0.html
  • Change to the folder of Keycloak
  • Set some common options via environment variables
    export JAVA_OPTS_APPEND="-javaagent:opentelemetry-javaagent_1_27_0.jar -Dotel.javaagent.debug=true"
    export OTEL_METRICS_EXPORTER=prometheus
    export KC_HEALTH_ENABLED=true
    export KC_METRICS_ENABLED=true
    
  • Start with HTTP only
    bin/kc.sh build
    eval exec bin/kc.sh start --hostname-strict=false --hostname-strict-https=false --http-enabled=true
    
  • Open http://localhost:8080/ to generate some metrics (the will be no metrics unless a page is being loaded)
  • Open http://localhost:9464/ to look at the metrics -> http_server_duration_milliseconds_count is present
  • Stop Keycloak by Ctrl+C on the CLI
  • create TLS key pair
  • Start Keycloak with TLS
    export KC_HTTPS_CERTIFICATE_FILE=tls.crt
    export KC_HTTPS_CERTIFICATE_KEY_FILE=tls.key
    bin/kc.sh build
    eval exec bin/kc.sh start --hostname-strict=false --hostname-strict-https=false
    
  • Open start page with TLS https://localhost:8443/
  • Open http://localhost:9464/ to look at the metrics -> http_server_duration_milliseconds_count is missing

What did you expect to see?

The metrics http_server_* should be visible both for TLS and non-TLS

What did you see instead?

The metrics http_server_* are missing for TLS and are only present in non-TLS

What version are you using?

1.27.0

Environment

$ java --version
openjdk 11.0.16.1 2022-08-12
OpenJDK Runtime Environment Temurin-11.0.16.1+1 (build 11.0.16.1+1)
OpenJDK 64-Bit Server VM Temurin-11.0.16.1+1 (build 11.0.16.1+1, mixed mode)
$ cat /etc/os-release 
NAME="Fedora Linux"
VERSION="38 (Workstation Edition)"

Additional context

I'm testing this to do performance tests on Keycloak for the upcoming version of Keycloak. I'm happy to support by testing this / running pre-releases / debugging etc.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the provided Keycloak 22.0.0 reproduction with the Java agent, comparing HTTP-only and TLS startup through the Netty HTTP instrumentation. Use the reported http_server_duration_milliseconds metrics as the check: the metrics should be present for both non-TLS and TLS HTTP/2 requests.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.