Graylog2 / Graylog2/graylog2-server

ClusterMetricsResource hard timeout for big clusters

Open
#17,100 1 comment 1 reaction 0 assignees View on GitHub
feature needs-discussion triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

We have a customer with very big cluster consisting of some 160 graylog nodes. The ClusterMetricsResource is constantly failing to deliver responses, because there is a hard limit configured:

```
Duration.ofMillis(Math.min(defaultCallTimeout.toMilliseconds(), 1000))
```

the nodes are all responding, but the 1000ms hard limit is just to strict to collect all metrics.

## Expected Behavior
ClusterMetricsResource doesn't fail for larger installations

## Possible Solution
Either make the 1000ms configurable as well (and strongly warn against changing that, because it can lead to exhausting thread pools elsewhere) or let the nodes send their metrics regularly, persist them and let ClusterMetricsResource just aggregate these results.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.