elastic / elastic/elastic-package
Add support for injecting configuration data into the transform definitions
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 72
- Forks
- 141
- Avg merge
- 19h 42m
- Merged PRs (30d)
- 55
Description
Currently integration configuration data, for example [`interval`](https://github.com/elastic/integrations/blob/main/packages/qualys_vmdr/data_stream/asset_host_detection/manifest.yml#L15), cannot be injected into the transform definition, confirmed [here](https://github.com/elastic/package-spec/issues/833#issuecomment-3249402300).
Making the transform retention user-configurable is often asked before, for example [here](https://github.com/elastic/enhancements/issues/26303). Some integrations have a workaround by injecting variables into input config, then into the ingest pipeline, and then into the transform through a field. This workaround is less efficient. It would be nice to control it better by directly injecting the configuration data into the transforms. Also a direct option to control the transform would have a better user experience as well.
By allowing user configuration data injected into the transform, users can control the transform retention efficiently.
Example: Based on [`interval`](https://github.com/elastic/integrations/blob/main/packages/qualys_vmdr/data_stream/asset_host_detection/manifest.yml#L15-L20) value,
```yaml
retention_policy:
time:
field: "@timestamp"
max_age: {{ interval }}
```
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the referenced packages/qualys_vmdr/data_stream/asset_host_detection/manifest.yml example and review the linked package-spec discussion. Trace how integration configuration currently reaches transform definitions and determine the required design. Done means users can inject configuration such as interval directly into a transform so it controls retention without the input-to-pipeline workaround.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- cli, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100