prometheus / prometheus/docs

document recommendations for reverse proxying prometheus

Open
#2,950 5 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
TypeScript
Stars
717
Forks
1.2k
Avg merge
3d 15h
Merged PRs (30d)
19

Description

It would be good if there were some recommendations documented about which settings to use when putting prometheus behind a reverse proxy like nginx.

I'm thinking of stuff like this:

proxy_set_header        Host $host;
proxy_set_header        X-Real-IP $remote_addr;
proxy_set_header        X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header        X-Forwarded-Proto $scheme;
proxy_set_header        X-Forwarded-Host $host;
proxy_set_header        X-Forwarded-Server $hostname;

(Haven't checked that this is a sensible assortment for prometheus specifically.

But then there is also the /api/v1/notifications/live endpoint which uses Server-sent events which requires special care so that is passes through nginx.

https://prometheus.io/docs/prometheus/latest/querying/api/#live-notifications

If this isn't configured correctly there will be a notification as such after a short while:

Real-time notifications interrupted.
Image showing Real-time notifications interrupted. Please refresh the page or check your connection.

With the following config it hasn't popped up yet. But i haven't checked how sensible every single one of those settings are:

upstream prometheus {
        server [::1]:9090 ;
}
server {
        listen 0.0.0.0:80 ;
        listen [::0]:80 ;
        server_name prometheus.example.org ;
        location / {
                return 301 https://$host$request_uri;
        }
}
server {
        listen 0.0.0.0:443 ssl ;
        listen [::0]:443 ssl ;
        server_name prometheus.example.org  ;
        http2 on;
        ssl_certificate /var/lib/acme/prometheus.example.org/fullchain.pem;
        ssl_certificate_key /var/lib/acme/prometheus.example.org/key.pem;
        ssl_trusted_certificate /var/lib/acme/prometheus.example.org/chain.pem;
        location / {
                proxy_pass http://prometheus;proxy_set_header
                # this is the code snippet in the top of the issue:
                include /nix/store/bjs5fy079yy18vqcfhzqw4011kc2rw04-nginx-recommended-proxy_set_header-headers.conf;
        }
        location =/api/v1/notifications/live {
                proxy_pass http://prometheus;
                proxy_http_version 1.1;
                proxy_set_header Connection "";
                proxy_buffering off;
                proxy_cache off;
                proxy_read_timeout 10m;
                proxy_send_timeout 10m;
                gzip off;
                # this is the code snippet in the top of the issue:
                include /nix/store/bjs5fy079yy18vqcfhzqw4011kc2rw04-nginx-recommended-proxy_set_header-headers.conf;
        }
}

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the Prometheus HTTP API live notifications page and the reverse-proxy configuration shown in the issue. Determine which nginx settings are required for normal requests and for SSE at /api/v1/notifications/live. Done means adding documentation with reliable reverse-proxy recommendations, including SSE handling that avoids the interruption warning.

Written by the indexing model from the issue text.

Assessment

Tech stack
nginx, prometheus
Domain
documentation
Issue type
Documentation
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.