Graylog2 / Graylog2/graylog2-server

Allow defining maximum size for index set instead of index count

Open
#9,282 3 comments 0 reactions 0 assignees View on GitHub
elasticsearch feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

Hi all

It seems like graylog index rotation by size doesn't work as it should.
I have an index set configured like so:

> Index rotation strategy:Index Size
> Max index size:10737418240 bytes (10.0GiB)
>

But indices never reaches 10Gb before rotating
A few samples:

> haproxy_945 Contains messages from 3 hours ago up to 29 minutes ago (7.7GiB / 8,759,567 messages) Show Details / Actions
> haproxy_944 Contains messages from 4 hours ago up to 3 hours ago (8.3GiB / 9,526,364 messages) Show Details / Actions
> haproxy_943 Contains messages from 5 hours ago up to 4 hours ago (7.2GiB / 8,319,854 messages) Show Details / Actions
> haproxy_942 Contains messages from 7 hours ago up to 5 hours ago (8.6GiB / 9,923,712 messages) Show Details / Actions
> haproxy_941 Contains messages from 7 hours ago up to 7 hours ago (7.2GiB / 8,286,737 messages) Show Details / Actions
> ...
> haproxy_915 Contains messages from a day ago up to a day ago (7.1GiB / 8,132,948 messages) Show Details / Actions
> haproxy_914 Contains messages from a day ago up to a day ago (6.8GiB / 7,872,737 messages) Show Details / Actions
> haproxy_913 Contains messages from a day ago up to a day ago (6.7GiB / 7,774,717 messages) Show Details / Actions
> haproxy_912 Contains messages from a day ago up to a day ago (6.6GiB / 7,678,127 messages) Show Details / Actions
> haproxy_911 Contains messages from a day ago up to a day ago (6.9GiB / 7,964,886 messages) Show Details / Actions
> haproxy_910 Contains messages from a day ago up to a day ago (6.9GiB / 7,955,319 messages) Show Details / Actions
>

As you can see, it never reaches even close to 10GB.
This is super problematic as I need to calculate the amount of storage allocated to the Elasticsearch by max index amount * size, but it's unreliable, and I end up losing some of the data (EG - I want to have 1Tib of data to have approximately the timeframe I need but it's rotated far before that).

This happened to me with graylog 2.4.4 and now also with 3.3.8. installed by RPM on amazon linux 2 server.
Elasticsearch cluster is an AWS ES cluster, version 6.8

any help would be appreciated

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.