opensearch-project / opensearch-project/data-prepper

Create a Robust CombineAggregateAction

Open
#855 4 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

backlog enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
The RFC for Stateful Aggregation provides an example of a basic CombineAggregateAction that creates an aggregated Event by simply using putAll, which will overwrite any existing keys of the combined Event.

Describe the solution you'd like
We should discuss a solution for providing a more robust CombineAggregateAction. Instead of simply overwriting keys that have merge conflicts, we can make the merging logic configurable for the user. For example, an additional configuration option for merging could look like this if the desire was to overwrite the keys:

processor:
  - aggregate:
      identification_keys:
        - 'sourceIp'
        - 'destinationIp'
        - 'port'
      window_duration: 180
      data_path: data/aggregate
      action:
        combine:
          merge_function: overwrite

And then if a user wanted to instead append to a list of values instead of overwrite, they could change the merge function to something like append, which would push the new conflicting key value to the end of a list

processor:
  - aggregate:
      identification_keys:
        - 'sourceIp'
        - 'destinationIp'
        - 'port'
      window_duration: 180
      data_path: data/aggregate
      action:
        combine:
          merge_function: append

Ideally, we can cover as many common merge functions as possible out of the box. However, we should also try to make it as easy as possible for the user to create their own custom merge functions.

Alternative Solution
The alternative to this solution is to simply make a completely new AggregateAction when a user does not wish to simply overwrite keys.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the RFC for Stateful Aggregation and its basic CombineAggregateAction example, then review the proposed aggregate configuration and merge_function options in this issue. Determine the supported conflict-merging behavior, including overwrite, append, and custom functions; done requires an agreed design for a robust CombineAggregateAction.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.