elastic / elastic/package-spec
[Change Proposal] First-level support of indexes for non-time-series
- Dominant language
- Go
- Stars
- 20
- Forks
- 93
- Avg merge
- 15h 10m
- Merged PRs (30d)
- 12
Description
Packages were designed for time series use cases, where the data is stored in data streams, with timestamps.
There are though some use cases that expect to store data that is not linked to any point in time. When using packages, these use cases rely now on Kibana plugins to configure what cannot be done from packages, like index management without data streams. This couples some package releases to Kibana releases, as they need to keep compatibility, and it doesn't allow the definition of field mappings and so on in the usual way for package development.
There is a supported workaround: using transforms. A package could configure an agent to send data to a data stream, and a transform would keep the most updated documents. This doesn't work for all use cases and is inefficient as it adds the additional data stream and the transform itself that otherwise wouldn't be necessary.
Support definition of indexes in integration and input packages, so they can be used the same way as data streams.
Some tasks that will be required:
* [ ] Check what bulk operations would be supported in Elastic Agent for each input and output. Data streams only support `create`, and this is probably the current default. Indexes probably need `index`/`delete`. From this assessment define the expected behavior for indexes, and if some toggle is needed depending on cases.
* [ ] Define how to declare indexes in the Package Spec. It should be ideally the same as data streams.
* [ ] Support index creation in Fleet.
* [ ] Make changes in elastic-agent/beats if needed for the different bulk operations.
Contributor guide
Assessment
This issue has not been assessed yet.