Would it be possible for Curator to cache its index metadata over chained tasks in a task file?
- Dominant language
- Python
- Stars
- 3.1k
- Forks
- 627
- PR merge metrics
- No merged PRs in 30d
Description
On very large clusters, it can take a few minutes for curator to generate its index metadata. I noticed in the debug logs that these two steps in particular took a very long time:
```
2016-10-26 16:14:12,504 DEBUG curator.indexlist _get_metadata:145 Getting index metadata
2016-10-26 16:14:12,504 DEBUG curator.indexlist empty_list_check:182 Checking for empty list
```
There is an obvious inefficiency here when your realize that my first task
```
1:
action: close
description: >-
close indices. We do this first to remove heap pressure on the warm nodes.
options:
continue_if_exception: True
ignore_empty_list: True
delete_aliases: True
timeout_override: 600
filters:
- filtertype: pattern
kind: prefix
value: 'logstash-'
exclude: False
- filtertype: age
source: creation_date
direction: older
unit: days
unit_count: 30
exclude: False
2:
action: delete_indices
description: >-
delete indices. We do this early on to free up space on the warm nodes
options:
continue_if_exception: True
ignore_empty_list: True
timeout_override: 600
filters:
- filtertype: pattern
kind: prefix
value: 'logstash-'
exclude: False
- filtertype: age
source: creation_date
direction: older
unit: days
unit_count: 90
exclude: False
```
yielded an empty list; the second necessarily will as well.
It's definitely possible that the status of indices may change as Curator tasks run. But it seems like Curator may benefit from the option of caching the index metadata between tasks, perhaps as a seperate "task" that generates all of the index metadata initially so it can be reused in subsequent tasks (as part of a "use_cached_metadata" option for example)
Contributor guide
Assessment
This issue has not been assessed yet.