elastic / elastic/package-spec

[Change Proposal] Stop using dot-prefixes for hidden indices & data streams

Open
#266 9 comments 1 reaction 0 assignees View on GitHub
discuss
Dominant language
Go
Stars
20
Forks
93
Avg merge
15h 10m
Merged PRs (30d)
12

Description

In Elasticsearch 7.7, support for an explicit hidden index setting was added (https://github.com/elastic/elasticsearch/pull/50452). This is preferred over the `.`-prefix that was traditionally used to indicate indices that were created by the system and should be excluded from API calls to Elasticsearch by default. Support for hidden data streams was added to the package-spec in https://github.com/elastic/package-spec/pull/91.

### Problem

Today, the Endpoint package ships hidden indices and data streams. It's worth noting that no other package currently uses this feature. Here are the assets currently using this:
- https://github.com/elastic/package-storage/blob/6db57140b24e4b0b815918834ca9917fa15637d5/packages/endpoint/1.4.1/data_stream/action_responses/manifest.yml#L4
- https://github.com/elastic/package-storage/blob/6db57140b24e4b0b815918834ca9917fa15637d5/packages/endpoint/1.4.1/data_stream/actions/manifest.yml#L4
- https://github.com/elastic/package-storage/blob/6db57140b24e4b0b815918834ca9917fa15637d5/packages/endpoint/1.4.1/data_stream/collection/manifest.yml#L4
- https://github.com/elastic/package-storage/blob/6db57140b24e4b0b815918834ca9917fa15637d5/packages/endpoint/1.4.1/elasticsearch/index_template/metrics-metadata-united.json#L531
- https://github.com/elastic/package-storage/blob/6db57140b24e4b0b815918834ca9917fa15637d5/packages/endpoint/1.4.1/elasticsearch/transform/metadata_united/default.json#L9

These indices violate the data stream naming scheme that we rely on in many places throughout the Stack and requires that we handle these data streams and indices as a special case. For example:
- For hidden data streams, we currently prepend a dot to all related data stream assets (index templates, component templates, pipeline names, etc.)
- This is handled in Kibana's Fleet plugin here: https://github.com/elastic/kibana/blob/0ee514b1fe044cdaca253f8f14988fc17635fed3/x-pack/plugins/fleet/server/services/epm/elasticsearch/index.ts#L10-L18
- We have to handle this as a special case in the elastic-package tooling https://github.com/elastic/elastic-package/pull/673
- We have to add privileges to `kibana_system` for these indices explicitly. There has been more than one case where forgetting to add these special cases have resulted in regressions. Example special cases:
- https://github.com/elastic/elasticsearch/blob/ad5a3fc81ee76465116cdff7ae3e1acfa22211e7/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStore.java#L751-L753
- https://github.com/elastic/elasticsearch/blob/ad5a3fc81ee76465116cdff7ae3e1acfa22211e7/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStore.java#L761
- https://github.com/elastic/elasticsearch/blob/ad5a3fc81ee76465116cdff7ae3e1acfa22211e7/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStore.java#L766
- https://github.com/elastic/elasticsearch/blob/ad5a3fc81ee76465116cdff7ae3e1acfa22211e7/x-pack/plugin/core/src/main/java/org/elasticsearch/xpack/core/security/authz/store/ReservedRolesStore.java#L777-L778

### Proposal

To reduce the amount of special handling we need for these indices, I propose that we:
- Change the package installation behavior to no longer prepend a `.` to these data stream asset names
- Add validation to the package-spec to disallow index names to be dot-prefixed in `elasticsearch/index_template` and destination indices in `elasticsearch/transform`
- Related: We don't currently seem to have anything [in our spec](https://github.com/elastic/package-spec/blob/main/versions/1/elasticsearch/spec.yml) for either of these types of resources?
- Remove special casing from elastic-package tooling and kibana_system's role descriptor

### Open Questions

- [ ] Can we make this change in a way that won't break pushing updates to the Endpoint package to older Stack versions? Does the Endpoint team need to be able to do this?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.