Graylog2 / Graylog2/graylog2-server

Load balanced multiple nodes return incorrect node metrics.

Open
#3,959 19 comments 0 reactions 0 assignees View on GitHub
to-verify triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

When multiple nodes are setup in a load balanced environment Graylog is confused as to which data belong to which nodes. The setup also uses SSL and certificates. Both the UI and REST ports are set the same.

Also see api/cluster/metrics/multiple Returns Inconsistent Results Nodes #3161
As the results are exactly as described in this defect.

## Expected Behavior

The call api/cluster/metrics/multiple should return the correct node data for each node in the returned json.

## Current Behavior

As it is difficult to recognize this issue, it is best seen on the System/logging page. Also it does not manifest itself all the time, so one must watch for it: The number of internal log messages for each node will jump around, and more telling is when one chooses a log level option from the drop down (when it is in this state), it will set it on a different node from the one that was chosen.

## Possible Solution

It is pretty clear the round robin-ing of the REST API port calls by the load balancer is causing the server, when it makes the api/cluster/metrics/multiple call, to get confused as to which data belongs to which.

Setting the system up exactly the same but using different REST/UI ports for each of the running Graylog nodes fixes the issue, however then of-course the API and UI calls are no longer load balanced.

The documentation is not clear as to how Graylog is suppose to figure out which rest call belongs to which node when they point at the same URL (we are using a cert) and point at the same port in an LB scenario, and clearly it doesn't figure it out, so this looks very much like a bug.

Since the data is returned correctly, but randomly not associated with the correct node, one can assume the rest calls are all working.

## Steps to Reproduce (for bugs)

1. With more than one graylog cluster behind a load balancer
2. Visit /system/logging
3. Observe changes in the log levels

## Context

Serious impact, the logging and node UI pages are largely non-functional.

## Your Environment

Graylog Version: 2.2.3
Elasticsearch Version: 2.4.1
MongoDB Version: 2.4.9
Operating System: Cent OS 7
Browser version: Chrome 59

Our cluster is setup with 3 Graylog Nodes behind Marathon LB [HAProxy] (test 1) or behind NGINX (test2)
Enabling stickiness made no difference.

Example nginx:

upstream graylog_web_interface {
server 192.168.117.17:6503;
server 192.168.14.200:6503;
server 192.168.96.185:6503;
}
server
{
listen 6503 ssl;
server_name example.local;

ssl_certificate /etc/nginx/dal10/cert.pem;
ssl_certificate_key /etc/nginx/dal10/pkcs8-plain.pem;

location /
{
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Graylog-Server-URL https://example.local:6503/api;
proxy_pass https://graylog_web_interface;
}
}

See : api/cluster/metrics/multiple Returns Inconsistent Results Nodes #3161 for further info.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.