Graylog2 / Graylog2/graylog2-server
warn about number of elasticsearch shards on index page
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
## What?
In Graylog when you create indices and the configuration of shards and replica it is easy to overwhelm the given resources and have in the end elasticsearch and with that Graylog not working proper.
The math of how many shards elasticsearch can handle is easy to make based on the tips in this elastic blog posting and this two tips quoted.
https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster
> **TIP:** Small shards result in small segments, which increases overhead. Aim to keep the average shard size between at least a few GB and a few tens of GB. For use-cases with time-based data, it is common to see shards between 20GB and 40GB in size.
>
> **TIP:** The number of shards you can hold on a node will be proportional to the amount of heap you have available, but there is no fixed limit enforced by Elasticsearch. A good rule-of-thumb is to ensure you keep the number of shards per node below 20 per GB heap it has configured. A node with a 30GB heap should therefore have a maximum of 600 shards, but the further below this limit you can keep it the better. This will generally help the cluster stay in good health.
This data can be read from the elasticsearch api and calculated in Graylog to generate a warning if the limits are over what elasticsearch can accomplish in the given limits.
## Why?
Most users do not know about those limitations and run over. This generate a burden and is a bad user experience. As Graylog does not warn about this limit Graylog takes the blame.
Giving a warning would create a better user experience and lower the support we need to give for that.
## Your Environment
* Graylog Version: 3.3 / 4.0
Contributor guide
Assessment
This issue has not been assessed yet.