elastic / elastic/apm-server

APM Data plugin asset installation feedback loop

Open
#14,811 6 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Go
Stars
1.3k
Forks
543
Avg merge
1d 18h
Merged PRs (30d)
109

Description

After 8.15.0, the APM assets are installed by the `apm-data` plugin in ES.
In case an error occurs, we've observed:
- Elasticsearch logs errors (good) as below
- It seems it retries continuously to install the assets (good, but maybe it should retry periodically?)

```
[instance-0000000054] error adding index template [traces-apm@template] for [apm] java.lang.IllegalArgumentException: composable template [traces-apm@template] template after composition with component templates [traces@mappings, apm@mappings, apm@settings, apm-10d@lifecycle, traces-apm@mappings, traces-apm-fallback@lifecycle, traces@custom, traces-apm@custom, ecs@mappings] is invalid at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.validateIndexTemplateV2(MetadataIndexTemplateService.java:753) ~[elasticsearch-8.15.5.jar:?] at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:640) ~[elasticsearch-8.15.5.jar:?] at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService.addIndexTemplateV2(MetadataIndexTemplateService.java:587) ~[elasticsearch-8.15.5.jar:?] at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$5.execute(MetadataIndexTemplateService.java:532) ~[elasticsearch-8.15.5.jar:?] at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$1.executeTask(MetadataIndexTemplateService.java:149) ~[elasticsearch-8.15.5.jar:?] at org.elasticsearch.cluster.metadata.MetadataIndexTemplateService$1.executeTask(MetadataIndexTemplateService.java:146) ~[elasticsearch-8.15.5.jar:?] at org.elasticsearch.cluster.SimpleBatchedExecutor.execute(SimpleBatchedExecutor.java:70) ~[elasticsearch-8.15.5.jar:?] at org.elasticsearch.cluster.service.MasterService.innerExecuteTasks(MasterService.java:1070) ~[elasticsearch-8.15.5.jar:?] at org.elasticsearch.cluster.service.MasterService.executeTasks(MasterService.java:1033) ~[elasticsearch-8.15.5.jar:?] at org.elasticsearch.cluster.service.MasterService.executeAndPublishBatch(MasterService.java:233) ~[elasticsearch-8.15.5.jar:?] at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.lambda$run$2(MasterService.java:1686) ~[elasticsearch-8.15.5.jar:?] at org.elasticsearch.action.ActionListener.run(ActionListener.java:444) ~[elasticsearch-8.15.5.jar:?] at org.elasticsearch.cluster.service.MasterService$BatchingTaskQueue$Processor.run(MasterService.java:1683) ~[elasti
```

In this particular case, it was due to the presence of a conflicting mapping between `traces-apm@custom` (by the user) and `traces-apm@mappings` (our, which changed between 8.12 and 8.15)

But the user has no feedback except if they are looking at the logs...
We should find a way to "expose this" to Kibana.

Other comments:
- Each APM data stream is handled singularly, so some assets installations might succeed, others might fail (e.g. `traces-apm@template` might be successful, but `traces-apm.rum@template` might fail)
- No way for users to "test" the setup prior installing the assets

Failure to install assets can generate data loss, so it's not something which can be left unaddressed.

FYI @simitt (not urgent but raised to keep track of it)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.