elastic / elastic/package-spec

[Change Proposal] Make transform source and destination configurable

Open
#386 0 comments 0 reactions 0 assignees View on GitHub
discuss Team:Ecosystem
Dominant language
Go
Stars
20
Forks
93
Avg merge
15h 10m
Merged PRs (30d)
12

Description

## Background
As of today, the source and destination indices of transforms need to be hard-coded into the transform itself. This means that if one needs to run the same transform on multiple data sources, they'd need to create a separate transform for each source.

For transforms shipped via Fleet, this poses the following challenges:
- Having multiple transform assets in the package, each corresponding to a different source index
- The number of input data sources the user might want to run the transform on is not definite
- The package will need to be updated each time a new data source need transformation

## Usecase
A user has data from different customers saved in different spaces. The user wants to run a transform on each of the data sources and store the results for each customer in separate destination indices as well.

## Proposal
There is precedence in Fleet to make source indices for assets such as unsupervised machine learning (ML) jobs [configurable](https://github.com/elastic/integrations/blob/main/packages/dga/kibana/ml_module/dga-ml.json#L62-L63) i.e. a user can choose which source index pattern(s) to run the ML jobs on, after the integration package has been installed.

Could this approach be extended to transforms where users can configure the source and destination indices after the fact? This would allow having a single transform asset in the integration package, which could be used on multiple different data sources.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.