elastic / elastic/elastic-package

Add support for injecting configuration data into the transform definitions

Open
#3,112 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Team:Ecosystem
Dominant language
Go
Stars
72
Forks
141
Avg merge
19h 42m
Merged PRs (30d)
55

Description

Currently integration configuration data, for example [`interval`](https://github.com/elastic/integrations/blob/main/packages/qualys_vmdr/data_stream/asset_host_detection/manifest.yml#L15), cannot be injected into the transform definition, confirmed [here](https://github.com/elastic/package-spec/issues/833#issuecomment-3249402300).

Making the transform retention user-configurable is often asked before, for example [here](https://github.com/elastic/enhancements/issues/26303). Some integrations have a workaround by injecting variables into input config, then into the ingest pipeline, and then into the transform through a field. This workaround is less efficient. It would be nice to control it better by directly injecting the configuration data into the transforms. Also a direct option to control the transform would have a better user experience as well.

By allowing user configuration data injected into the transform, users can control the transform retention efficiently.

Example: Based on [`interval`](https://github.com/elastic/integrations/blob/main/packages/qualys_vmdr/data_stream/asset_host_detection/manifest.yml#L15-L20) value,

```yaml
retention_policy:
time:
field: "@timestamp"
max_age: {{ interval }}
```

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the referenced packages/qualys_vmdr/data_stream/asset_host_detection/manifest.yml example and review the linked package-spec discussion. Trace how integration configuration currently reaches transform definitions and determine the required design. Done means users can inject configuration such as interval directly into a transform so it controls retention without the input-to-pipeline workaround.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
cli, tooling
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.