Graylog2 / Graylog2/graylog2-server

In 6.2, Cluster Metric API calls to other nodes time out after 1s ignoring config

Open
#22,595 0 comments 0 reactions 0 assignees View on GitHub
bug triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

A community post [here](https://community.graylog.org/t/inter-node-api-timeout-in-graylog-6-2-0-multi-region-cluster/35521) raises the following:

*

Graylog 6.2.0 cluster with nodes distributed across multiple geographical regions. When I log in to a node in one region, unable to view performance metrics (e.g., Memory/Heap, Buffers, Journal) for nodes in other regions. However, if I log in to a node in the same geographic location, all metrics display correctly.

In the logs seeing inter-node API timeouts like this:

```
2025-04-29T18:30:10.070Z WARN [ProxiedResource] Failed to call API on node <1ce8335f-e3a9-4d66-b1eb-4bdcdedf827b>, cause: timeout (duration: 1002 ms)
2025-04-29T18:30:10.070Z WARN [ProxiedResource] Failed to call API on node <09ac9ab1-ec2c-4e54-88a0-1c74f0291dfe>, cause: timeout (duration: 1001 ms)
2025-04-29T18:30:10.070Z WARN [ProxiedResource] Failed to call API on node , cause: timeout (duration: 1001 ms)
2025-04-29T18:30:10.070Z WARN [ProxiedResource] Failed to call API on node <1db2c9da-8c1c-4f24-9b6e-b85f49fadd94>, cause: timeout (duration: 1002 ms)
2025-04-29T18:31:10.071Z WARN [ProxiedResource] Failed to call API on node , cause: timeout (duration: 1001 ms)
2025-04-29T18:31:10.071Z WARN [ProxiedResource] Failed to call API on node <1ce8335f-e3a9-4d66-b1eb-4bdcdedf827b>, cause: timeout (duration: 1001 ms)
2025-04-29T18:31:10.071Z WARN [ProxiedResource] Failed to call API on node <1db2c9da-8c1c-4f24-9b6e-b85f49fadd94>, cause: timeout (duration: 1001 ms)
2025-04-29T18:31:10.071Z WARN [ProxiedResource] Failed to call API on node <09ac9ab1-ec2c-4e54-88a0-1c74f0291dfe>, cause: timeout (duration: 1001 ms)
2025-04-29T18:31:10.071Z WARN [ProxiedResource] Failed to call API on node , cause: timeout (duration: 1001 ms)
2025-04-29T18:31:10.071Z WARN [ProxiedResource] Failed to call API on node <7569ace1-6ea9-4f67-8531-da9c0919ee3d>, cause: timeout (duration: 1002 ms)
2025-04-29T18:31:58.277Z WARN [ProxiedResource] Failed to call API on node , cause: timeout (duration: 1001 ms)
2025-04-29T18:31:58.277Z WARN [ProxiedResource] Failed to call API on node , cause: timeout (duration: 1001 ms)
2025-04-29T18:31:58.277Z WARN [ProxiedResource] Failed to call API on node <1ce8335f-e3a9-4d66-b1eb-4bdcdedf827b>, cause: timeout (duration: 1001 ms)
2025-04-29T18:31:58.277Z WARN [ProxiedResource] Failed to call API on node <1db2c9da-8c1c-4f24-9b6e-b85f49fadd94>, cause: timeout (duration: 1001 ms)
```

tried uncommenting and setting proxied_requests_default_call_timeout = 5s in server.conf, but it doesn’t seem to have any effect—the timeouts still occur at around 1 second. also reviewed the config file but couldn’t find any other relevant settings to adjust this timeout.

Everything else in the cluster appears to be functioning properly. This issue is specifically with viewing node performance stats across regions.

Environment Details:

OS: Ubuntu 24.04
Graylog Version: 6.2.0
Number of Graylog nodes: 9 (will be scaling to 20+)
OpenSearch Version: 2.15.0
Number of OpenSearch data nodes: 32

I’ve noticed that if you visit a node’s metrics page e.g., https://10.X.X.X:9000/system/metrics/node/, it fails to load for any node located in a different geographic region than the one you’re currently accessing. This doesn’t appear to be a firewall or DNS issue—I’ve verified both are working correctly.

It’s also worth noting that this behavior was not present in Graylog 6.1.11. In that version, the cluster page displayed journal and JVM heap metrics for all nodes, which relied on successful metric queries across the cluster. It seems that something changed in Graylog 6.2 that affects this functionality.

Is there a new or alternative setting in Graylog 6.2.0 to increase the inter-node API call timeout, or is this a regression?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.