Graylog2 / Graylog2/graylog2-server

improved elasticsearch index management (ILM)

Open
#9,828 7 comments 1 reaction 1 assignee Claimed by @kroepke View on GitHub
feature triaged
Dominant language
Java
Stars
8.1k
Forks
1.1k
Avg merge
1d 20h
Merged PRs (30d)
217

Description

## What?
Have a more flexible elasticsearch managment and allow hot/warm configurations from within the Graylog UI.

Modern Elasticsearch versions would allow to write ILM configurations ( https://www.elastic.co/guide/en/elasticsearch/reference/current/overview-index-lifecycle-management.html ) like

```
PUT _ilm/policy/graylog_ilm
{
"policy": {
"phases": {
"hot": {
"min_age": "0ms",
"actions": {
"set_priority": {
"priority": 100
}
}
},
"warm": {
"min_age": "8d",
"actions": {
"forcemerge": {
"max_num_segments": 1
},
"set_priority": {
"priority": 50
},
"shrink": {
"number_of_shards": 4
}
}
},
"cold": {
"min_age": "32d",
"actions": {
"freeze": {}
}
}
}
}
}

```

Making use of the given abilities in Elasticsearch would make it easier to implement such a feature and allows user better control of their data storage.

## Why?
It was possible in the past (prior 4.0) to use Elasticsearch ILM ( https://www.elastic.co/guide/en/elasticsearch/reference/current/overview-index-lifecycle-management.html ) to maintain a hot warm cluster. Now the only left solution is to us curator with a configuration like this: https://gist.github.com/jalogisch/12d7e173c555505854a4f9bffbf3155d

This would eliminate the described way we have here: https://www.graylog.org/post/back-to-basics-using-a-hot-warm-elasticsearch-cluster

## Your Environment

* Graylog Version: 4.0
* Elasticsearch Version: 6/7

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.