elastic / elastic/integrations
[Usability] Improve/align user experience for Custom * Integrations
- Dominant language
- Handlebars
- Stars
- 333
- Forks
- 647
- Avg merge
- 3d 4h
- Merged PRs (30d)
- 209
Description
### Custom Logs integration in 2.0.0 onwards
Newly created Custom Logs integration policies added to a Fleet policy on 2.0.0 onwards will:
- Create an index template named `logs-` followed by the `dataset` name (e.g. `logs-mycustomdataset`)
- The Index template will include by default 2 component templates:
- `logs-` followed by the `dataset` name, followed by `@package` (e.g. `logs-mycustomdataset@package`), which contains the "basic" ECS fields for the Logs data streams - do not edit this policy as it can be overwritten on upgrades
- `logs-` followed by the `dataset` name, followed by `@custom` one (e.g. `logs-mycustomdataset@custom`), which is empty by default and can be used by the final user to customize the mappings and settings of the data stream - it will not be overwritten on upgrades even if the metadata says `_meta.managed: true`
The `@package` component template will also include a `default_pipeline` named `logs-`, followed by the `dataset` name, followed by `-` (e.g. `logs-mycustomdataset-2.0.0`).
If you need to run an ingest pipeline, just define a custom pipeline named `logs-`, followed by the `dataset` name, followed by `@custom` (e.g. `logs-mycustoimdataset@custom` ingest pipeline), as it will be called automatically from the `default_pipeline` mentioned above.
### Custom Logs integration prior 2.0.0
In previous versions, the Custom Logs integration was not creating any index template automatically and all the setup was delegated to the final user.
Note Custom Logs integration policies installed in previous versions will not be changed when upgrading to 2.0.0 onwards to not generate any breaking change.
### Other Custom integrations
The Custom TCP, UDP, HTTP Endpoint... integrations do not behave in the same way as Custom Logs.
Those integrations will install a default index template named `logs-.generic` (e.g. `logs-tcp.generic`, `logs-http_endpoint.generic`, etc...).
If you edit the `dataset` name in the Fleet UI when adding the integration to a policy, it will be used only to define the destination data stream but not for the creation of the index templates.
---
It would be ideal to guide users to create a guided experience when using Custom Integrations.
See this discuss post https://discuss.elastic.co/t/split-a-data-stream-into-separate-indexes-to-allow-different-ilm-policies/365223/3 for a further example/confusion with Custom UDP logs
Contributor guide
Assessment
This issue has not been assessed yet.