elastic / elastic/fleet-server
@package component templates missing _tier_preference after logs@settings v23 upgrade
- Dominant language
- Go
- Stars
- 113
- Forks
- 117
- Avg merge
- 1d 16h
- Merged PRs (30d)
- 112
Description
## Problem Statement
Fleet-server's `@package` component templates (e.g., `logs-fleet_server.output_health@package`) do not set `_tier_preference`. Prior to a recent ES upgrade, the `logs@settings` built-in component template (now v23) explicitly set `data_hot,data_content` tier preference, ensuring new backing indices landed on hot-tier nodes with promotable shards. That setting was removed from `logs@settings` as part of Elastic's shift to Data Stream Lifecycle. Fleet-server's templates were not updated to compensate, leaving tier allocation to ES defaults. On affected deployments, new backing indices after a rollover land on nodes with unpromotable shards, which — combined with the missing `maintenance` privilege — causes 403s on output health writes.
This affects any customer on the ES version shipping `logs@settings` v23 whose fleet-server data streams have rolled over.
## Goals
- New fleet-server data stream backing indices are consistently allocated to the hot tier on standard deployments
- The fix survives future rollovers without manual intervention
## Non-Goals
- Fixing the missing `maintenance` privilege in the service account role (tracked [here](https://github.com/elastic/fleet-server/issues/7674))
- Modifying tier routing for non-fleet data streams
- Changing DSL lifecycle configuration
## Acceptance Criteria
- [ ] `logs-fleet_server.*` and `metrics-fleet_server.*` `@package` component templates include `_tier_preference: data_hot,data_content`
- [ ] After a rollover, new backing indices are allocated to the hot tier on a standard ESS deployment
- [ ] The fix does not interfere with DSL-managed lifecycle (`data_retention: 7d`) on the output health data stream
- [ ] Serverless deployments are unaffected (tier preference is a no-op in serverless)
Spike / Investigation Notes
Confirmed via `GET /_component_template/logs-fleet_server.output_health@package` that no `_tier_preference` is set. The `logs@settings` built-in template (`modified_date_millis` diverges from `created_date_millis`, consistent with an ES upgrade) no longer carries this setting. The immediate workaround applied to the affected customer was adding `_tier_preference` to the `logs-fleet_server.output_health@custom` component template (the `@custom` slot is listed in `ignore_missing_component_templates` and safe to populate). The upstream fix belongs in the `@package` templates so it ships with the integration rather than requiring per-deployment remediation.
## Open Questions
- Do `metrics-fleet_server.*` `@package` templates have the same gap? Spot-checked `logs-fleet_server.output_health` only during the incident.
## References
- [Issue #7674](https://github.com/elastic/fleet-server/issues/7674)
Contributor guide
Research direction
Locate the `logs-fleet_server.*` and `metrics-fleet_server.*` `@package` component templates and inspect their rendered settings with the component template API. Compare them with the `logs@settings` behavior and verify a rollover places new backing indices on the hot tier while preserving the 7d data retention and leaving serverless unaffected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elasticsearch
- Domain
- backend, search
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100