cloudflare / cloudflare/cloudflared
🐛 Missing prometheus metrics
- Dominant language
- Go
- Stars
- 15.6k
- Forks
- 1.4k
- PR merge metrics
- No merged PRs in 30d
Description
**Describe the bug**
I have been using cloudflared tunnels for a while now, and am very happy with the experience. I have started bringing in some metrics to Prometheus for Grafana to view as a dashboard. I have been able to use quite a lot of the provided metrics, but a few that I believe should be active are not showing up for me at all.
In particular, I am looking for the metric `cloudflared_tunnel_user_hostnames_counts`. I can see that it's listed as an available metric in [October 2020](https://github.com/cloudflare/cloudflared/issues/244) and again in [June 2022](https://github.com/cloudflare/cloudflared/issues/677). I can see the metric is created [here](https://github.com/cloudflare/cloudflared/blob/master/connection/metrics.go) and the logic hasn't been edited since [November 2020](https://github.com/cloudflare/cloudflared/commit/9ac40dcf04a5b05572051ec2f4c276d848bc8457). However, the metric does not show in the output of any of my machines.
**To Reproduce**
Machine 1:
```
justin@celebrimbor:~$ sudo journalctl -b -u cloudflared | tail -n50
[...]
Dec 15 11:38:05 celebrimbor systemd[1]: Starting cloudflared...
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Starting tunnel tunnelID=9cb1055b-b6b3-4331-bd47-5008bec287b9
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Version 2022.11.1
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF GOOS: linux, GOVersion: go1.19.3, GoArch: amd64
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Settings: map[metrics:0.0.0.0:8754 token:*****]
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF cloudflared will not automatically update if installed by a package manager.
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Generated Connector ID: 63b24db6-5322-4fa6-9307-69bd8f7c142f
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Will be fetching remotely managed configuration from Cloudflare API. Defaulting to protocol: quic
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Initial protocol quic
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF ICMP proxy will use 192.168.1.13 as source for IPv4
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF ICMP proxy will use 2600:1700:4559:8430::1cb in zone enp5s0 as source for IPv6
Dec 15 11:38:05 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:05Z INF Starting metrics server on [::]:8754/metrics
Dec 15 11:38:06 celebrimbor cloudflared[3022141]: 2022/12/15 11:38:06 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Dec 15 11:38:06 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:06Z INF Connection 8f5e238f-fef0-4f0d-93ed-2423d788571c registered with protocol: quic connIndex=0 ip=198.41.200.13 location=ORD
Dec 15 11:38:06 celebrimbor systemd[1]: Started cloudflared.
Dec 15 11:38:06 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:06Z INF Connection 66bb17f6-93bf-4267-8244-0c4cf23d0eb1 registered with protocol: quic connIndex=1 ip=198.41.192.77 location=MSP
Dec 15 11:38:07 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:07Z INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"calibre.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8083\"},{\"hostname\":\"prowlarr.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:9696\"},{\"hostname\":\"syncthing.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8384\"},{\"hostname\":\"hass.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8123\"},{\"hostname\":\"hassio.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8123\"},{\"hostname\":\"radarr.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:7878\"},{\"hostname\":\"code-c.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8443\"},{\"hostname\":\"sonarr.[redacted]\",\"service\":\"http://localhost:8989\"},{\"hostname\":\"rtorrent.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8085\"},{\"hostname\":\"vnc-c.[redacted]\",\"originRequest\":{},\"service\":\"rdp://localhost:5901\"},{\"hostname\":\"maxifold.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:9025\"},{\"hostname\":\"kbt.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:1313\"},{\"hostname\":\"nextcloud.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4431\"},{\"hostname\":\"manifold.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:3000\"},{\"hostname\":\"plex.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:32400\"},{\"service\":\"http_status:404\"}],\"warp-routing\":{\"enabled\":false}}" version=37
Dec 15 11:38:07 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:07Z INF Connection 9407f2dd-33f2-44ad-a595-f532f4cfc2c4 registered with protocol: quic connIndex=2 ip=198.41.200.23 location=ORD
Dec 15 11:38:08 celebrimbor cloudflared[3022141]: 2022-12-15T16:38:08Z INF Connection 9773cdfc-5650-4ec3-a8ce-976037df6013 registered with protocol: quic connIndex=3 ip=198.41.192.27 location=MSP
justin@celebrimbor:~$ curl -s localhost:8754/metrics | grep "# HELP cloudflared_"
# HELP cloudflared_config_local_config_pushes Number of local configuration pushes to the edge
# HELP cloudflared_config_local_config_pushes_errors Number of errors occurred during local configuration pushes
# HELP cloudflared_orchestration_config_version Configuration Version
# HELP cloudflared_tunnel_active_streams Number of active streams created by all muxers.
# HELP cloudflared_tunnel_concurrent_requests_per_tunnel Concurrent requests proxied through each tunnel
# HELP cloudflared_tunnel_ha_connections Number of active ha connections
# HELP cloudflared_tunnel_request_errors Count of error proxying to origin
# HELP cloudflared_tunnel_response_by_code Count of responses by HTTP status code
# HELP cloudflared_tunnel_server_locations Where each tunnel is connected to. 1 means current location, 0 means previous locations.
# HELP cloudflared_tunnel_timer_retries Unacknowledged heart beats count
# HELP cloudflared_tunnel_total_requests Amount of requests proxied through all the tunnels
# HELP cloudflared_tunnel_tunnel_authenticate_success Count of successful tunnel authenticate
# HELP cloudflared_tunnel_tunnel_register_success Count of successful tunnel registrations
justin@celebrimbor:~$ curl -s localhost:8754/metrics | grep "host"
justin@celebrimbor:~$
```
Machine 2:
```
justin@ereinion:~$ justin@ereinion:~$ sudo journalctl -b -u cloudflared | tail -n50
[...]
Dec 15 11:49:01 ereinion systemd[1]: Starting cloudflared...
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Starting tunnel tunnelID=078f38bd-3a0c-44ab-bd01-d9c3eee26071
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Version 2022.11.1
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF GOOS: linux, GOVersion: go1.19.3, GoArch: amd64
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Settings: map[metrics:0.0.0.0:8754 no-autoupdate:true token:*****]
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF cloudflared will not automatically update if installed by a package manager.
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Generated Connector ID: d6fb3312-7108-4f35-a410-a7e1f3a4860d
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Will be fetching remotely managed configuration from Cloudflare API. Defaulting to protocol: quic
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Initial protocol quic
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF ICMP proxy will use 167.99.56.166 as source for IPv4
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF ICMP proxy will use 2604:a880:800:10::6c2:a001 in zone eth0 as source for IPv6
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Starting metrics server on [::]:8754/metrics
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022/12/15 11:49:02 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Connection 4821df16-d264-4830-a090-33d48a97c7b5 registered with protocol: quic connIndex=0 ip=198.41.200.23 location=ORD
Dec 15 11:49:02 ereinion systemd[1]: Started cloudflared.
Dec 15 11:49:02 ereinion cloudflared[3165002]: 2022-12-15T16:49:02Z INF Connection 10bf319f-3ff2-4c72-a5ff-30c6154947f4 registered with protocol: quic connIndex=1 ip=198.41.192.7 location=EWR
Dec 15 11:49:03 ereinion cloudflared[3165002]: 2022-12-15T16:49:03Z INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"prometheus-old.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:9090\"},{\"hostname\":\"grafana-old.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:3000\"},{\"hostname\":\"element.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8183\"},{\"hostname\":\"kirkwall.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:30000\"},{\"hostname\":\"requests.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:5055\"},{\"hostname\":\"jupyter.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8888\"},{\"hostname\":\"paperless-old.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8000\"},{\"hostname\":\"synapse.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8008\"},{\"hostname\":\"code-e.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8443\"},{\"service\":\"http_status:404\"}],\"warp-routing\":{\"enabled\":false}}" version=84
Dec 15 11:49:03 ereinion cloudflared[3165002]: 2022-12-15T16:49:03Z INF Connection da5abd30-597f-4af5-9e49-8d78e801c1cd registered with protocol: quic connIndex=2 ip=198.41.200.73 location=ORD
Dec 15 11:49:04 ereinion cloudflared[3165002]: 2022-12-15T16:49:04Z INF Connection 44d3b9ca-4878-4951-93e2-38e33095e12e registered with protocol: quic connIndex=3 ip=198.41.192.77 location=EWR
justin@ereinion:~$ curl -s localhost:8754/metrics | grep "# HELP cloudflared_"
# HELP cloudflared_config_local_config_pushes Number of local configuration pushes to the edge
# HELP cloudflared_config_local_config_pushes_errors Number of errors occurred during local configuration pushes
# HELP cloudflared_orchestration_config_version Configuration Version
# HELP cloudflared_tunnel_active_streams Number of active streams created by all muxers.
# HELP cloudflared_tunnel_concurrent_requests_per_tunnel Concurrent requests proxied through each tunnel
# HELP cloudflared_tunnel_ha_connections Number of active ha connections
# HELP cloudflared_tunnel_request_errors Count of error proxying to origin
# HELP cloudflared_tunnel_response_by_code Count of responses by HTTP status code
# HELP cloudflared_tunnel_server_locations Where each tunnel is connected to. 1 means current location, 0 means previous locations.
# HELP cloudflared_tunnel_timer_retries Unacknowledged heart beats count
# HELP cloudflared_tunnel_total_requests Amount of requests proxied through all the tunnels
# HELP cloudflared_tunnel_tunnel_authenticate_success Count of successful tunnel authenticate
# HELP cloudflared_tunnel_tunnel_register_success Count of successful tunnel registrations
justin@ereinion:~$ curl -s localhost:8754/metrics | grep "host"
justin@ereinion:~$
```
Machine 3:
```
justin@kalak:~$ sudo journalctl -b -u cloudflared | tail -n50
[...]
Dec 15 11:53:27 kalak systemd[1]: Starting cloudflared...
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Starting tunnel tunnelID=0749b8a6-9268-42bf-9164-715e6caa8efc
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Cannot determine default configuration path. No file [config.yml config.yaml] in [~/.cloudflared ~/.cloudflare-warp ~/cloudflare-warp /etc/cloudflared /usr/local/etc/cloudflared]
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Version 2022.11.1
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF GOOS: linux, GOVersion: go1.19.3, GoArch: amd64
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Settings: map[edge-ip-version:6 metrics:0.0.0.0:8754 token:*****]
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF cloudflared will not automatically update if installed by a package manager.
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Generated Connector ID: 7316437a-b41a-45cb-bbca-afd4a1594ddd
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Will be fetching remotely managed configuration from Cloudflare API. Defaulting to protocol: quic
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Initial protocol quic
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF ICMP proxy will use 0.0.0.0 as source for IPv4
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF ICMP proxy will use ::1 in zone lo as source for IPv6
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 0 is not between ping group 1 to 0"
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 0 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Starting metrics server on [::]:8754/metrics
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022/12/15 11:53:27 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
Dec 15 11:53:27 kalak cloudflared[3725871]: 2022-12-15T16:53:27Z INF Connection a891c90e-7c5d-4405-b516-9bc3f43f45da registered with protocol: quic connIndex=0 ip=2606:4700:a8::8 location=VIE
Dec 15 11:53:27 kalak systemd[1]: Started cloudflared.
Dec 15 11:53:28 kalak cloudflared[3725871]: 2022-12-15T16:53:28Z INF Connection 52169dff-9e1a-4271-ae1f-840a3b5752cb registered with protocol: quic connIndex=1 ip=2606:4700:a0::5 location=FRA
Dec 15 11:53:28 kalak cloudflared[3725871]: 2022-12-15T16:53:28Z INF Updated to new configuration config="{\"ingress\":[{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:4080\"},{\"hostname\":\"code-k.[redacted]\",\"service\":\"http://localhost:8443\"},{\"hostname\":\"kirkwall-beta.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:30000\"},{\"hostname\":\"prometheus.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:9090\"},{\"hostname\":\"stream.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8084\"},{\"hostname\":\"grafana.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:3000\"},{\"hostname\":\"tautulli.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8181\"},{\"hostname\":\"requests-beta.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:5055\"},{\"hostname\":\"rss.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:8524\"},{\"hostname\":\"rss-bridge.[redacted]\",\"originRequest\":{},\"service\":\"http://localhost:3060\"},{\"service\":\"http_status:404\"}],\"warp-routing\":{\"enabled\":false}}" version=35
Dec 15 11:53:29 kalak cloudflared[3725871]: 2022-12-15T16:53:29Z INF Connection f78f61cb-6fb6-4898-865b-137dac49eb0d registered with protocol: quic connIndex=2 ip=2606:4700:a8::1 location=VIE
Dec 15 11:53:30 kalak cloudflared[3725871]: 2022-12-15T16:53:30Z INF Connection 502bc161-5036-47bd-9db8-6f2f6150de7b registered with protocol: quic connIndex=3 ip=2606:4700:a0::2 location=FRA
Dec 15 11:54:50 kalak cloudflared[3725871]: 2022-12-15T16:54:50Z ERR error="Incoming request ended abruptly: context canceled" cfRay=77a0b415be9603fc-ORD ingressRule=8 originService=http://localhost:8443
Dec 15 11:54:50 kalak cloudflared[3725871]: 2022-12-15T16:54:50Z ERR Request failed error="Incoming request ended abruptly: context canceled" connIndex=1 dest=https://code-k.[redacted]/stable-6261075646f055b99068d3688932416f2346dd3b/vscode-remote-resource?path=%2Fapp%2Fcode-server%2Flib%2Fvscode%2Fextensions%2Ftheme-kimbie-dark%2Fthemes%2Fkimbie-dark-color-theme.json&tkn= ip=2606:4700:a0::5 type=http
Dec 15 11:54:51 kalak cloudflared[3725871]: 2022-12-15T16:54:51Z ERR Failed to handle QUIC stream error=EOF connIndex=1 ip=2606:4700:a0::5
justin@kalak:~$ curl -s localhost:8754/metrics | grep "# HELP cloudflared_"
# HELP cloudflared_config_local_config_pushes Number of local configuration pushes to the edge
# HELP cloudflared_config_local_config_pushes_errors Number of errors occurred during local configuration pushes
# HELP cloudflared_orchestration_config_version Configuration Version
# HELP cloudflared_tunnel_active_streams Number of active streams created by all muxers.
# HELP cloudflared_tunnel_concurrent_requests_per_tunnel Concurrent requests proxied through each tunnel
# HELP cloudflared_tunnel_ha_connections Number of active ha connections
# HELP cloudflared_tunnel_request_errors Count of error proxying to origin
# HELP cloudflared_tunnel_response_by_code Count of responses by HTTP status code
# HELP cloudflared_tunnel_server_locations Where each tunnel is connected to. 1 means current location, 0 means previous locations.
# HELP cloudflared_tunnel_timer_retries Unacknowledged heart beats count
# HELP cloudflared_tunnel_total_requests Amount of requests proxied through all the tunnels
# HELP cloudflared_tunnel_tunnel_authenticate_success Count of successful tunnel authenticate
# HELP cloudflared_tunnel_tunnel_register_success Count of successful tunnel registrations
justin@kalak:~$ curl -s localhost:8754/metrics | grep "host"
justin@kalak:~$
```
All 3 of the above were configured via the dashboard. To see if this was a dashboard-specific issue, I started a tunnel via the local configuration:
```
justin@kalak:~$ cloudflared tunnel --config /opt/cloudflared/kalak.yml --metrics 0.0.0.0:8755 --edge-ip-version 6 run
2022-12-15T17:01:14Z INF Starting tunnel tunnelID=6c72ed50-4f6e-498a-a8a5-283891dcfc61
2022-12-15T17:01:14Z INF Version 2022.11.1
2022-12-15T17:01:14Z INF GOOS: linux, GOVersion: go1.19.3, GoArch: amd64
2022-12-15T17:01:14Z INF Settings: map[config:/opt/cloudflared/kalak.yml cred-file:/home/justin/.cloudflared/6c72ed50-4f6e-498a-a8a5-283891dcfc61.json credentials-file:/home/justin/.cloudflared/6c72ed50-4f6e-498a-a8a5-283891dcfc61.json edge-ip-version:6 metrics:0.0.0.0:8755]
2022-12-15T17:01:14Z INF cloudflared will not automatically update if installed by a package manager.
2022-12-15T17:01:14Z INF Generated Connector ID: a360f7f2-b2ab-454d-b631-c208d7707e19
2022-12-15T17:01:14Z INF Initial protocol quic
2022-12-15T17:01:14Z INF ICMP proxy will use 0.0.0.0 as source for IPv4
2022-12-15T17:01:14Z INF ICMP proxy will use ::1 in zone lo as source for IPv6
2022-12-15T17:01:14Z WRN The user running cloudflared process has a GID (group ID) that is not within ping_group_range. You might need to add that user to a group within that range, or instead update the range to encompass a group the user is already in by modifying /proc/sys/net/ipv4/ping_group_range. Otherwise cloudflared will not be able to ping this network error="Group ID 1000 is not between ping group 1 to 0"
2022-12-15T17:01:14Z WRN ICMP proxy feature is disabled error="cannot create ICMPv4 proxy: Group ID 1000 is not between ping group 1 to 0 nor ICMPv6 proxy: socket: permission denied"
2022-12-15T17:01:14Z INF Starting Hello World server at 127.0.0.1:43443
2022-12-15T17:01:14Z INF Starting metrics server on [::]:8755/metrics
2022/12/15 12:01:14 failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.
2022-12-15T17:01:15Z INF Connection 110621e7-7bde-48f3-866d-7c2ab504fc7d registered with protocol: quic connIndex=0 ip=2606:4700:a8::6 location=VIE
2022-12-15T17:01:16Z INF Connection c9267e82-1c8d-4454-8eed-07967c58be71 registered with protocol: quic connIndex=1 ip=2606:4700:a0::8 location=FRA
2022-12-15T17:01:17Z INF Connection 17df35be-a741-4620-b04b-5b0fe8a92e9d registered with protocol: quic connIndex=2 ip=2606:4700:a8::8 location=VIE
2022-12-15T17:01:18Z INF Connection 39463562-97a1-433a-b587-4f30109271ae registered with protocol: quic connIndex=3 ip=2606:4700:a0::1 location=FRA
[leave running...]
justin@kalak:~$ curl -s localhost:8755/metrics | grep "# HELP cloudflared_"
# HELP cloudflared_config_local_config_pushes Number of local configuration pushes to the edge
# HELP cloudflared_config_local_config_pushes_errors Number of errors occurred during local configuration pushes
# HELP cloudflared_orchestration_config_version Configuration Version
# HELP cloudflared_tunnel_active_streams Number of active streams created by all muxers.
# HELP cloudflared_tunnel_concurrent_requests_per_tunnel Concurrent requests proxied through each tunnel
# HELP cloudflared_tunnel_ha_connections Number of active ha connections
# HELP cloudflared_tunnel_request_errors Count of error proxying to origin
# HELP cloudflared_tunnel_response_by_code Count of responses by HTTP status code
# HELP cloudflared_tunnel_server_locations Where each tunnel is connected to. 1 means current location, 0 means previous locations.
# HELP cloudflared_tunnel_timer_retries Unacknowledged heart beats count
# HELP cloudflared_tunnel_total_requests Amount of requests proxied through all the tunnels
# HELP cloudflared_tunnel_tunnel_authenticate_success Count of successful tunnel authenticate
# HELP cloudflared_tunnel_tunnel_register_success Count of successful tunnel registrations
justin@kalak:~$ curl -s localhost:8755/metrics | grep "host"
justin@kalak:~$
```
My assumption is that this metric is only enabled under specific circumstances, but I cannot tell what those would be. I may be missing some configuration option to make this metric available or some part of this setup may be incompatible with the metric I'm looking for.
**Expected behavior**
I expect the metric `cloudflared_tunnel_user_hostnames_counts` (or a metric with a similar function) to be available for use with Prometheus.
**Environment and versions**
- OS: `Ubuntu 20.04.5 LTS`, `Ubuntu 20.04.5 LTS`, `Debian GNU/Linux 11 (bullseye)`
- Architecture: `x86_64`, `x86_64`, `x86_64`
- Version:
- `cloudflared version 2022.11.1 (built 2022-11-30-1048 UTC)`
- `cloudflared version 2022.11.1 (built 2022-11-30-1048 UTC)`
- `cloudflared version 2022.11.1 (built 2022-11-30-1048 UTC)`
**Logs and errors**
See above
**Additional context**
N/A
Contributor guide
Assessment
This issue has not been assessed yet.