elastic / elastic/integrations

[Elasticsearch]: transform logs-elasticsearch.index_pivot-default should exclude frozen tier

Open
#17,722 2 comments 1 reaction 0 assignees View on GitHub
Integration:elasticsearch needs:triage Team:Stack Monitoring
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

### Integration Name

Elasticsearch [elasticsearch]

### Dataset Name

elasticsearch.index, elasticsearch.stack_monitoring.index

### Integration Version

1.19.1

### Agent Version

9.2.5

### OS Version and Architecture

ECH

### User Goal

The logs-elasticsearch.index_pivot-default-0.3.0 needs to avoide searching the Frozen tier.

### Existing Features

Transform performance is severely impacted by reading data from the frozen tier. This results in a growing delay between current being data written to the monitoring indices and the transforms output index. In my case because of retention policies the lag was over 6 months.

Updating the transforms DSL filter to include "must not" logic will allow the search to avoid the frozen tier. This greatly improved search performance and allows the transform to keep pace with the ingest rate of the monitoring indices.

```
"filter": [
{
"exists": {
"field": "elasticsearch.index.creation_date"
}
},
{
"exists": {
"field": "elasticsearch.index.tier_preference"
}
},
{
"bool": {
"must_not": {
"terms": {
"_tier": [
"data_frozen"
]
}
}
}
}
],
```

### What did you see?

Transform Job lags behind monitoring index ingest rate when querying data from the frozen tier.

### Anything else?

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.