elastic / elastic/integrations
[Elasticsearch]: Add retention_policy to index_pivot transform
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
## Summary
Add a transform `retention_policy` to the Elasticsearch integration's `index_pivot` transform so the Fleet-managed `monitoring-indices` destination does not grow without bound. Transform retention is a **Technical Preview** feature in Elasticsearch; this issue tracks the integrations-side change.
## Problem
The `index_pivot` transform (`logs-elasticsearch.index_pivot-default-{VERSION}`) in `packages/elasticsearch/elasticsearch/transform/index_pivot/transform.yml` writes aggregated index consumption metrics to a flat `monitoring-indices` destination. The transform defines no `retention_policy` today (`fleet_transform_version` is `0.4.0`).
Package documentation states that `monitoring-indices` is not controlled by index lifecycle management (ILM) and that you must [activate ILM on the target index](https://www.elastic.co/guide/en/elasticsearch/reference/current/getting-started-index-lifecycle-management.html#manage-time-series-data-without-data-streams) if you need retention on clusters with many indices or long retention. That manual step is easy to miss and contributes to unbounded disk use for the consumption workflow.
## Proposed solution
1. Add `retention_policy` to `packages/elasticsearch/elasticsearch/transform/index_pivot/transform.yml`, aligned with other integrations that use transform retention (for example, `time.field: "@timestamp"` and a `max_age` value). A **90d** default is a reasonable starting point for consumption metrics, but the Stack Monitoring team should confirm the right window for product and support expectations.
2. Bump `_meta.fleet_transform_version` so Fleet reinstalls and restarts the transform when the package upgrades.
3. Update `packages/elasticsearch/docs/README.md` (and generated docs if applicable) to describe automatic transform retention, note the Technical Preview status, and clarify when you still need ILM or other tuning on `monitoring-indices`.
4. Add a changelog entry and validate against a stack version that supports transform `retention_policy` on pivot transforms.
## Related work
- [#15054](https://github.com/elastic/integrations/issues/15054) — `monitoring-indices` is created without an alias or ILM; documents operational impact (for example, large index growth). This issue focuses on transform-level retention via `retention_policy` and does not replace alias or ILM improvements tracked there.
- [elastic/elasticsearch#144012](https://github.com/elastic/elasticsearch/pull/144012) — related Elasticsearch transform work (append-only destination writes). Useful if a future design moves the destination to a data stream with ILM instead of transform `retention_policy`.
## Acceptance criteria
- [ ] `retention_policy` is defined on the `index_pivot` transform with `@timestamp` as the time field and a team-agreed `max_age`.
- [ ] `fleet_transform_version` is incremented so existing deployments pick up the change on upgrade.
- [ ] README documents default retention behavior, Technical Preview status, and relationship to manual ILM on `monitoring-indices`.
- [ ] Changelog entry is added for the Elasticsearch integration package.
- [ ] Transform installs and runs successfully on a stack version that supports transform `retention_policy`, with verified expiry of data older than `max_age`.
Contributor guide
Research direction
Start with packages/elasticsearch/elasticsearch/transform/index_pivot/transform.yml and compare retention_policy usage in other integrations. Check the package versioning and changelog conventions, then update packages/elasticsearch/docs/README.md and validate installation and expiry on a stack version supporting pivot retention. Done means the agreed @timestamp and max_age are applied, the fleet_transform_version is bumped, documentation and changelog are updated, and the transform runs successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch
- Domain
- databases, documentation
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100