open-telemetry / open-telemetry/opentelemetry-java-instrumentation
HTTP Instrumentation on Quarkus/Netty doesn't work for TLS + HTTP/2
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 2.6k
- Forks
- 1.2k
- Avg merge
- 2d 18h
- Merged PRs (30d)
- 228
Description
Describe the bug
When I start Keycloak 21.1.2 or 22.0.0 (which is based on Quarkus and Netty) with TLS and HTTP/2, I don't get http_server_duration_milliseconds* metrics.
When I start it with HTTP only (which will fallback to HTTP/1.1), I get them:
http_server_duration_milliseconds_count{otel_scope_name="io.opentelemetry.netty-4.1",otel_scope_version="1.27.0-alpha",http_method="GET",http_route="/",http_scheme="http",http_status_code="200",net_host_name="localhost",net_host_port="8080",net_protocol_name="http",net_protocol_version="1.1"} 1.0 1689095194824
Steps to reproduce
- Download Keycloak 22.0.0 and extract - https://www.keycloak.org/archive/downloads-22.0.0.html
- Change to the folder of Keycloak
- Set some common options via environment variables
export JAVA_OPTS_APPEND="-javaagent:opentelemetry-javaagent_1_27_0.jar -Dotel.javaagent.debug=true" export OTEL_METRICS_EXPORTER=prometheus export KC_HEALTH_ENABLED=true export KC_METRICS_ENABLED=true - Start with HTTP only
bin/kc.sh build eval exec bin/kc.sh start --hostname-strict=false --hostname-strict-https=false --http-enabled=true - Open
http://localhost:8080/to generate some metrics (the will be no metrics unless a page is being loaded) - Open
http://localhost:9464/to look at the metrics ->http_server_duration_milliseconds_countis present - Stop Keycloak by Ctrl+C on the CLI
- create TLS key pair
- Start Keycloak with TLS
export KC_HTTPS_CERTIFICATE_FILE=tls.crt export KC_HTTPS_CERTIFICATE_KEY_FILE=tls.key bin/kc.sh build eval exec bin/kc.sh start --hostname-strict=false --hostname-strict-https=false - Open start page with TLS
https://localhost:8443/ - Open
http://localhost:9464/to look at the metrics ->http_server_duration_milliseconds_countis missing
What did you expect to see?
The metrics http_server_* should be visible both for TLS and non-TLS
What did you see instead?
The metrics http_server_* are missing for TLS and are only present in non-TLS
What version are you using?
1.27.0
Environment
$ java --version
openjdk 11.0.16.1 2022-08-12
OpenJDK Runtime Environment Temurin-11.0.16.1+1 (build 11.0.16.1+1)
OpenJDK 64-Bit Server VM Temurin-11.0.16.1+1 (build 11.0.16.1+1, mixed mode)
$ cat /etc/os-release
NAME="Fedora Linux"
VERSION="38 (Workstation Edition)"
Additional context
I'm testing this to do performance tests on Keycloak for the upcoming version of Keycloak. I'm happy to support by testing this / running pre-releases / debugging etc.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run the provided Keycloak 22.0.0 reproduction with the Java agent, comparing HTTP-only and TLS startup through the Netty HTTP instrumentation. Use the reported http_server_duration_milliseconds metrics as the check: the metrics should be present for both non-TLS and TLS HTTP/2 requests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100