Graylog2 / Graylog2/graylog2-server
Load balanced multiple nodes return incorrect node metrics.
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
When multiple nodes are setup in a load balanced environment Graylog is confused as to which data belong to which nodes. The setup also uses SSL and certificates. Both the UI and REST ports are set the same.
Also see api/cluster/metrics/multiple Returns Inconsistent Results Nodes #3161
As the results are exactly as described in this defect.
## Expected Behavior
The call api/cluster/metrics/multiple should return the correct node data for each node in the returned json.
## Current Behavior
As it is difficult to recognize this issue, it is best seen on the System/logging page. Also it does not manifest itself all the time, so one must watch for it: The number of internal log messages for each node will jump around, and more telling is when one chooses a log level option from the drop down (when it is in this state), it will set it on a different node from the one that was chosen.
## Possible Solution
It is pretty clear the round robin-ing of the REST API port calls by the load balancer is causing the server, when it makes the api/cluster/metrics/multiple call, to get confused as to which data belongs to which.
Setting the system up exactly the same but using different REST/UI ports for each of the running Graylog nodes fixes the issue, however then of-course the API and UI calls are no longer load balanced.
The documentation is not clear as to how Graylog is suppose to figure out which rest call belongs to which node when they point at the same URL (we are using a cert) and point at the same port in an LB scenario, and clearly it doesn't figure it out, so this looks very much like a bug.
Since the data is returned correctly, but randomly not associated with the correct node, one can assume the rest calls are all working.
## Steps to Reproduce (for bugs)
1. With more than one graylog cluster behind a load balancer
2. Visit /system/logging
3. Observe changes in the log levels
## Context
Serious impact, the logging and node UI pages are largely non-functional.
## Your Environment
Graylog Version: 2.2.3
Elasticsearch Version: 2.4.1
MongoDB Version: 2.4.9
Operating System: Cent OS 7
Browser version: Chrome 59
Our cluster is setup with 3 Graylog Nodes behind Marathon LB [HAProxy] (test 1) or behind NGINX (test2)
Enabling stickiness made no difference.
Example nginx:
upstream graylog_web_interface {
server 192.168.117.17:6503;
server 192.168.14.200:6503;
server 192.168.96.185:6503;
}
server
{
listen 6503 ssl;
server_name example.local;
ssl_certificate /etc/nginx/dal10/cert.pem;
ssl_certificate_key /etc/nginx/dal10/pkcs8-plain.pem;
location /
{
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Graylog-Server-URL https://example.local:6503/api;
proxy_pass https://graylog_web_interface;
}
}
See : api/cluster/metrics/multiple Returns Inconsistent Results Nodes #3161 for further info.
Contributor guide
Assessment
This issue has not been assessed yet.