Graylog2 / Graylog2/graylog2-server

Prometheus exporter improvements

Open
#10,810 2 comments 0 reactions 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

We added a basic Prometheus exporter in https://github.com/Graylog2/graylog2-server/pull/10767. This issue collects a few improvement ideas for it.

## File based service discovery endpoint

Prometheus supports different service discovery methods to discover scrape targets dynamically. One of those methods is [file-based service discovery](https://prometheus.io/docs/guides/file-sd/). This method watches a JSON file in the file system that contains the scrape targets. The file will be reloaded when it changes.

It might be nice if we could offer an API endpoint in Graylog that returns this JSON structure including all active Graylog nodes that have the Prometheus exporter enabled. (incl. IP address and exporter port)

An example use case for this would be a cronjob that runs on Prometheus nodes and polls the API endpoint, writing the response to a file that will be monitored and reloaded by the Prometheus process using the file-based service discovery.

## TLS & Authentication

The current Prometheus exporter endpoint doesn't support TLS and authentication. This would be a great improvement from a security standpoint. Prometheus supports both for scrape targets: https://prometheus.io/docs/prometheus/latest/configuration/configuration/#scrape_config

Any secrets like passwords or access tokens that we add to the Graylog configuration for this need to be stored encrypted or hashed!

## Include JVM metrics

The Prometheus [Java instrumentation library](https://github.com/prometheus/client_java) we are using is able to export some generic JVM metrics. We might want to offer users to enable these so they are included in the exporter metrics.

See: https://github.com/prometheus/client_java/tree/master/simpleclient_hotspot/src/main/java/io/prometheus/client/hotspot

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.