Graylog2 / Graylog2/graylog2-server
Allow defining maximum size for index set instead of index count
- Dominant language
- Java
- Stars
- 8.1k
- Forks
- 1.1k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 217
Description
Hi all
It seems like graylog index rotation by size doesn't work as it should.
I have an index set configured like so:
> Index rotation strategy:Index Size
> Max index size:10737418240 bytes (10.0GiB)
>
But indices never reaches 10Gb before rotating
A few samples:
> haproxy_945 Contains messages from 3 hours ago up to 29 minutes ago (7.7GiB / 8,759,567 messages) Show Details / Actions
> haproxy_944 Contains messages from 4 hours ago up to 3 hours ago (8.3GiB / 9,526,364 messages) Show Details / Actions
> haproxy_943 Contains messages from 5 hours ago up to 4 hours ago (7.2GiB / 8,319,854 messages) Show Details / Actions
> haproxy_942 Contains messages from 7 hours ago up to 5 hours ago (8.6GiB / 9,923,712 messages) Show Details / Actions
> haproxy_941 Contains messages from 7 hours ago up to 7 hours ago (7.2GiB / 8,286,737 messages) Show Details / Actions
> ...
> haproxy_915 Contains messages from a day ago up to a day ago (7.1GiB / 8,132,948 messages) Show Details / Actions
> haproxy_914 Contains messages from a day ago up to a day ago (6.8GiB / 7,872,737 messages) Show Details / Actions
> haproxy_913 Contains messages from a day ago up to a day ago (6.7GiB / 7,774,717 messages) Show Details / Actions
> haproxy_912 Contains messages from a day ago up to a day ago (6.6GiB / 7,678,127 messages) Show Details / Actions
> haproxy_911 Contains messages from a day ago up to a day ago (6.9GiB / 7,964,886 messages) Show Details / Actions
> haproxy_910 Contains messages from a day ago up to a day ago (6.9GiB / 7,955,319 messages) Show Details / Actions
>
As you can see, it never reaches even close to 10GB.
This is super problematic as I need to calculate the amount of storage allocated to the Elasticsearch by max index amount * size, but it's unreliable, and I end up losing some of the data (EG - I want to have 1Tib of data to have approximately the timeframe I need but it's rotated far before that).
This happened to me with graylog 2.4.4 and now also with 3.3.8. installed by RPM on amazon linux 2 server.
Elasticsearch cluster is an AWS ES cluster, version 6.8
any help would be appreciated
Contributor guide
Assessment
This issue has not been assessed yet.