elastic / elastic/fleet-server

@package component templates missing _tier_preference after logs@settings v23 upgrade

Open
#7,675 0 comments 0 reactions 0 assignees View on GitHub
bug Team:Elastic-Agent-Control-Plane
Dominant language
Go
Stars
113
Forks
117
Avg merge
1d 16h
Merged PRs (30d)
112

Description

## Problem Statement

Fleet-server's `@package` component templates (e.g., `logs-fleet_server.output_health@package`) do not set `_tier_preference`. Prior to a recent ES upgrade, the `logs@settings` built-in component template (now v23) explicitly set `data_hot,data_content` tier preference, ensuring new backing indices landed on hot-tier nodes with promotable shards. That setting was removed from `logs@settings` as part of Elastic's shift to Data Stream Lifecycle. Fleet-server's templates were not updated to compensate, leaving tier allocation to ES defaults. On affected deployments, new backing indices after a rollover land on nodes with unpromotable shards, which — combined with the missing `maintenance` privilege — causes 403s on output health writes.

This affects any customer on the ES version shipping `logs@settings` v23 whose fleet-server data streams have rolled over.

## Goals

- New fleet-server data stream backing indices are consistently allocated to the hot tier on standard deployments
- The fix survives future rollovers without manual intervention

## Non-Goals

- Fixing the missing `maintenance` privilege in the service account role (tracked [here](https://github.com/elastic/fleet-server/issues/7674))
- Modifying tier routing for non-fleet data streams
- Changing DSL lifecycle configuration

## Acceptance Criteria

- [ ] `logs-fleet_server.*` and `metrics-fleet_server.*` `@package` component templates include `_tier_preference: data_hot,data_content`
- [ ] After a rollover, new backing indices are allocated to the hot tier on a standard ESS deployment
- [ ] The fix does not interfere with DSL-managed lifecycle (`data_retention: 7d`) on the output health data stream
- [ ] Serverless deployments are unaffected (tier preference is a no-op in serverless)

Spike / Investigation Notes

Confirmed via `GET /_component_template/logs-fleet_server.output_health@package` that no `_tier_preference` is set. The `logs@settings` built-in template (`modified_date_millis` diverges from `created_date_millis`, consistent with an ES upgrade) no longer carries this setting. The immediate workaround applied to the affected customer was adding `_tier_preference` to the `logs-fleet_server.output_health@custom` component template (the `@custom` slot is listed in `ignore_missing_component_templates` and safe to populate). The upstream fix belongs in the `@package` templates so it ships with the integration rather than requiring per-deployment remediation.

## Open Questions

- Do `metrics-fleet_server.*` `@package` templates have the same gap? Spot-checked `logs-fleet_server.output_health` only during the incident.

## References

- [Issue #7674](https://github.com/elastic/fleet-server/issues/7674)

Contributor guide

Open the contributing guide

Research direction

Locate the `logs-fleet_server.*` and `metrics-fleet_server.*` `@package` component templates and inspect their rendered settings with the component template API. Compare them with the `logs@settings` behavior and verify a rollover places new backing indices on the hot tier while preserving the 7d data retention and leaving serverless unaffected.

Written by the indexing model from the issue text.

Assessment

Tech stack
elasticsearch
Domain
backend, search
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.