Graylog2 / Graylog2/graylog2-server

Allow index level settings to be manipulated during index creating / rotation (read: storage management)

Open
#1,555 4 comments 1 reaction 0 assignees View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

... as discussed with @kroepke this issue is mainly for reference ...

Because of our load & data size we can no longer treat all our ES data nodes equally and need to implement a hot / warm architecture (keeping the youngest indices on really fast machines and migrate them to slower boxes as they get older).

The currently preferred way of doing this is to manipulate the 'index.routing.allocation.require.*' setting.

In order to do this efficiently we need support for that in our graylog-server nodes so I would like to have 3 configuration options :
- one option that defines the tag name
- one option that defines the tag content for new indicies
- one option that defines the tag content for rotated indices

An example could be :
elasticsearch_index_routing_allocation_tag_name = machine_type
elasticsearch_index_routing_allocation_new_index_tag = fast
elasticsearch_index_routing_allocation_old_index_tag = bulk

This would set 'index.routing.allocation.require.machine_type' to the value 'fast' for new indices and then change that to 'bulk' whenever an index get's rotated.

Our current retention strategy would allow us to have today's index on fast machines and have ES migrate them as they get rotated - for maximum flexibility one would need another configuration option to define how may indices to keep with the elasticsearch_index_routing_allocation_new_index_tag setting.

An example could be :
elasticsearch_index_routing_allocation_new_index_count = 5

This would ensure that the 5 newest indices are kept with ''index.routing.allocation.require.machine_type' set to 'fast'.

Background info :
- https://www.elastic.co/guide/en/elasticsearch/guide/current/retiring-data.html#migrate-indices
- https://www.elastic.co/blog/hot-warm-architecture

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.