Graylog2 / Graylog2/graylog2-server
Prometheus exporter improvements
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
We added a basic Prometheus exporter in https://github.com/Graylog2/graylog2-server/pull/10767. This issue collects a few improvement ideas for it.
## File based service discovery endpoint
Prometheus supports different service discovery methods to discover scrape targets dynamically. One of those methods is [file-based service discovery](https://prometheus.io/docs/guides/file-sd/). This method watches a JSON file in the file system that contains the scrape targets. The file will be reloaded when it changes.
It might be nice if we could offer an API endpoint in Graylog that returns this JSON structure including all active Graylog nodes that have the Prometheus exporter enabled. (incl. IP address and exporter port)
An example use case for this would be a cronjob that runs on Prometheus nodes and polls the API endpoint, writing the response to a file that will be monitored and reloaded by the Prometheus process using the file-based service discovery.
## TLS & Authentication
The current Prometheus exporter endpoint doesn't support TLS and authentication. This would be a great improvement from a security standpoint. Prometheus supports both for scrape targets: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config
Any secrets like passwords or access tokens that we add to the Graylog configuration for this need to be stored encrypted or hashed!
## Include JVM metrics
The Prometheus [Java instrumentation library](https://github.com/prometheus/client_java) we are using is able to export some generic JVM metrics. We might want to offer users to enable these so they are included in the exporter metrics.
See: https://github.com/prometheus/client_java/tree/master/simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot
Contributor guide
Assessment
This issue has not been assessed yet.