elastic / elastic/apm-server

Kibana does not remove index template for removed data streams on integrations upgrade

Open
#13,181 5 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
1.3k
Forks
543
Avg merge
1d 18h
Merged PRs (30d)
109

Description

When we will [migrate](https://github.com/elastic/apm-server/issues/11529) from `apm` integration package to `apm-data` ES plugin we will need to trigger a removal of index templates created by the integration, which by default are created with a higher priority than the one created by the ES plugin.

To ensure this steps works I run a test to verify if Kibana removes index template when upgrading an integration where installed data streams have been removed. I tested these steps:
1. setup a Es+Kibana stack and install `apm` integration package (tested with `8.15.0-SNAPSHOT` but any current version should lead to same results)
2. check installed Index Templates
3. install a custom version of `apm` integration package where all data streams except one (`traces` - due to how `elastic-package` works is not possible to package an integration without at least 1 data stream; this is not relevant for this test but should be investigated to proceed further with the migration) have been removed
4. Verify if Index Templates for removed data streams have been removed.

The test showed that Index Templates are not removed. This is a blocker for https://github.com/elastic/apm-server/issues/11529 as templates from the plugin would be overridden by the integration package templates.

Assets difference between packages

Assets with normal integration package:
![2024-05-16_16-06](https://github.com/elastic/apm-server/assets/526307/1c366629-20be-4577-a4a3-a3311fa387d1)

Assets with custom integration package:
![2024-05-16_16-08](https://github.com/elastic/apm-server/assets/526307/63fcaafd-7a17-4ede-b7f1-9ac19c9dfb3a)

Index templates before installing custom version

![image](https://github.com/elastic/apm-server/assets/526307/e750453b-6490-4835-a1b5-2abe070d3932)
![image](https://github.com/elastic/apm-server/assets/526307/a6c92c14-5d8e-46d1-ba28-fb618e00eee2)

Index templates after installing custom version

![Screenshot 2024-05-16 at 16-09-02 Index Management - Elastic](https://github.com/elastic/apm-server/assets/526307/ad77b604-cfec-4119-a51d-0f26f30f3ef0)
![Screenshot 2024-05-16 at 16-09-10 Index Management - Elastic](https://github.com/elastic/apm-server/assets/526307/49f9c80c-15be-42c8-a777-bcc771409890)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.