elastic / elastic/integrations

[Elasticsearch]: Add retention_policy to index_pivot transform

Open
#19,714 1 comment 1 reaction 0 assignees View on GitHub
Integration:elasticsearch Team:Stack Monitoring
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

## Summary

Add a transform `retention_policy` to the Elasticsearch integration's `index_pivot` transform so the Fleet-managed `monitoring-indices` destination does not grow without bound. Transform retention is a **Technical Preview** feature in Elasticsearch; this issue tracks the integrations-side change.

## Problem

The `index_pivot` transform (`logs-elasticsearch.index_pivot-default-{VERSION}`) in `packages/elasticsearch/elasticsearch/transform/index_pivot/transform.yml` writes aggregated index consumption metrics to a flat `monitoring-indices` destination. The transform defines no `retention_policy` today (`fleet_transform_version` is `0.4.0`).

Package documentation states that `monitoring-indices` is not controlled by index lifecycle management (ILM) and that you must [activate ILM on the target index](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#manage-time-series-data-without-data-streams) if you need retention on clusters with many indices or long retention. That manual step is easy to miss and contributes to unbounded disk use for the consumption workflow.

## Proposed solution

1. Add `retention_policy` to `packages/elasticsearch/elasticsearch/transform/index_pivot/transform.yml`, aligned with other integrations that use transform retention (for example, `time.field: "@timestamp"` and a `max_age` value). A **90d** default is a reasonable starting point for consumption metrics, but the Stack Monitoring team should confirm the right window for product and support expectations.
2. Bump `_meta.fleet_transform_version` so Fleet reinstalls and restarts the transform when the package upgrades.
3. Update `packages/elasticsearch/docs/README.md` (and generated docs if applicable) to describe automatic transform retention, note the Technical Preview status, and clarify when you still need ILM or other tuning on `monitoring-indices`.
4. Add a changelog entry and validate against a stack version that supports transform `retention_policy` on pivot transforms.

## Related work

- [#15054](https://github.com/elastic/integrations/issues/15054) — `monitoring-indices` is created without an alias or ILM; documents operational impact (for example, large index growth). This issue focuses on transform-level retention via `retention_policy` and does not replace alias or ILM improvements tracked there.
- [elastic/elasticsearch#144012](https://github.com/elastic/elasticsearch/pull/144012) — related Elasticsearch transform work (append-only destination writes). Useful if a future design moves the destination to a data stream with ILM instead of transform `retention_policy`.

## Acceptance criteria

- [ ] `retention_policy` is defined on the `index_pivot` transform with `@timestamp` as the time field and a team-agreed `max_age`.
- [ ] `fleet_transform_version` is incremented so existing deployments pick up the change on upgrade.
- [ ] README documents default retention behavior, Technical Preview status, and relationship to manual ILM on `monitoring-indices`.
- [ ] Changelog entry is added for the Elasticsearch integration package.
- [ ] Transform installs and runs successfully on a stack version that supports transform `retention_policy`, with verified expiry of data older than `max_age`.

Contributor guide

Open the contributing guide

Research direction

Start with packages/elasticsearch/elasticsearch/transform/index_pivot/transform.yml and compare retention_policy usage in other integrations. Check the package versioning and changelog conventions, then update packages/elasticsearch/docs/README.md and validate installation and expiry on a stack version supporting pivot retention. Done means the agreed @timestamp and max_age are applied, the fleet_transform_version is bumped, documentation and changelog are updated, and the transform runs successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
databases, documentation
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.