elastic / elastic/integrations
[Elasticsearch]: transform logs-elasticsearch.index_pivot-default should exclude frozen tier
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
### Integration Name
Elasticsearch [elasticsearch]
### Dataset Name
elasticsearch.index, elasticsearch.stack_monitoring.index
### Integration Version
1.19.1
### Agent Version
9.2.5
### OS Version and Architecture
ECH
### User Goal
The logs-elasticsearch.index_pivot-default-0.3.0 needs to avoide searching the Frozen tier.
### Existing Features
Transform performance is severely impacted by reading data from the frozen tier. This results in a growing delay between current being data written to the monitoring indices and the transforms output index. In my case because of retention policies the lag was over 6 months.
Updating the transforms DSL filter to include "must not" logic will allow the search to avoid the frozen tier. This greatly improved search performance and allows the transform to keep pace with the ingest rate of the monitoring indices.
```
"filter": [
{
"exists": {
"field": "elasticsearch.index.creation_date"
}
},
{
"exists": {
"field": "elasticsearch.index.tier_preference"
}
},
{
"bool": {
"must_not": {
"terms": {
"_tier": [
"data_frozen"
]
}
}
}
}
],
```
### What did you see?
Transform Job lags behind monitoring index ingest rate when querying data from the frozen tier.
### Anything else?
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.