opensearch-project / opensearch-project/data-prepper
Create a Robust CombineAggregateAction
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
The RFC for Stateful Aggregation provides an example of a basic CombineAggregateAction that creates an aggregated Event by simply using putAll, which will overwrite any existing keys of the combined Event.
Describe the solution you'd like
We should discuss a solution for providing a more robust CombineAggregateAction. Instead of simply overwriting keys that have merge conflicts, we can make the merging logic configurable for the user. For example, an additional configuration option for merging could look like this if the desire was to overwrite the keys:
processor:
- aggregate:
identification_keys:
- 'sourceIp'
- 'destinationIp'
- 'port'
window_duration: 180
data_path: data/aggregate
action:
combine:
merge_function: overwrite
And then if a user wanted to instead append to a list of values instead of overwrite, they could change the merge function to something like append, which would push the new conflicting key value to the end of a list
processor:
- aggregate:
identification_keys:
- 'sourceIp'
- 'destinationIp'
- 'port'
window_duration: 180
data_path: data/aggregate
action:
combine:
merge_function: append
Ideally, we can cover as many common merge functions as possible out of the box. However, we should also try to make it as easy as possible for the user to create their own custom merge functions.
Alternative Solution
The alternative to this solution is to simply make a completely new AggregateAction when a user does not wish to simply overwrite keys.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the RFC for Stateful Aggregation and its basic CombineAggregateAction example, then review the proposed aggregate configuration and merge_function options in this issue. Determine the supported conflict-merging behavior, including overwrite, append, and custom functions; done requires an agreed design for a robust CombineAggregateAction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100