HTTP2 not working

Host operating system: output of uname -a

Linux ip-172-16-209-7 5.10.0-11-cloud-amd64 #1 SMP Debian 5.10.92-1 (2022-01-18) x86_64 GNU/Linux

node_exporter version: output of node_exporter --version

node_exporter, version 1.3.1 (branch: HEAD, revision: a2321e7b940ddcff26873612bccdf7cd4c42b6b6)
  build user:       root@243aafa5525c
  build date:       20211205-11:09:49
  go version:       go1.17.3
  platform:         linux/amd64

node_exporter command line flags

--web.listen-address=0.0.0.0:9100 --log.level=debug --web.config=/etc/prometheus/exporters/web.yml

Are you running node_exporter in Docker?

No

What did you do that produced an error?

As per docs I am using a web.conf file, which looks like this:

http_server_config:
    http2: true
tls_server_config:
    cert_file: /etc/prometheus/exporters/node_exporter.crt
    key_file: /etc/prometheus/exporters/node_exporter.key

This does work, but not on HTTP2:

curl -Ivk https://localhost:9100/metrics
*   Trying 127.0.0.1:9100...
* Connected to localhost (127.0.0.1) port 9100 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: CN=ip-172-16-209-7.eu-central-1.compute.internal prometheus node exporter
*  start date: Feb  9 21:31:32 2022 GMT
*  expire date: Feb  7 21:31:32 2032 GMT
*  issuer: CN=ip-172-16-209-7.eu-central-1.compute.internal prometheus node exporter
*  SSL certificate verify result: self signed certificate (18), continuing anyway.
> HEAD /metrics HTTP/1.1
> Host: localhost:9100
> User-Agent: curl/7.74.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< Content-Type: text/plain; version=0.0.4; charset=utf-8
Content-Type: text/plain; version=0.0.4; charset=utf-8
< Date: Wed, 09 Feb 2022 21:42:12 GMT
Date: Wed, 09 Feb 2022 21:42:12 GMT

< 
* Connection #0 to host localhost left intact

FYI, this is my curl:

curl 7.74.0 (x86_64-pc-linux-gnu) libcurl/7.74.0 OpenSSL/1.1.1k zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh2/1.9.0 nghttp2/1.43.0 librtmp/2.3
Release-Date: 2020-12-09
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps mqtt pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: alt-svc AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets

What did you expect to see?

A response over HTTP2

What did you see instead?

A response over HTTP/1.1

Start with the node_exporter web.config entry point and the provided http_server_config and tls_server_config settings, then reproduce the request with curl against /metrics. Investigate why ALPN does not negotiate h2; done means the same HTTPS endpoint returns a response over HTTP/2 when http2 is enabled.