opensearch-project / opensearch-project/data-prepper

Support moving nested json to event root

Open
#3,377 2 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
The KeyValue processor writes its output maps to a destination field that is not the root of the event. As far as I'm aware there is not a way to move a nested field like that to the root of the event.

Example event

{message: "key1=val1 key2=val2"}

Current behavior of KeyValue processor

{
   message: "key1=val1 key2=val2",
   parsed_message: {
       key1: "val1",
       key2: "val2"
   }

}

Desired output event

{
   message: "key1=val1 key2=val2",
   key1: "val1",
   key2: "val2"
}

There are other cases beyond the KeyValue Processor where moving a nested field to the event root so building it into a processor would be useful

Describe the solution you'd like
The CopyValue processor should provide an optional parameter like to_root which can copy a field to the event root. It should also support a way of moving all fields within a json object

The configuration to get the desired output from above could look something like this:

  processor:
    - copy_values:
        entries:
        - from_object_key: "parsed_message"
          to_root: true

Describe alternatives you've considered (Optional)
Data Prepper could support something like a Flatten Object processor to flatten nested json.

That config might look something like this

  processor:
    - flatten:
        - key: "parsed_message"

And change the input to be like this

{
   message: "key1=val1 key2=val2",
   parsed_message.key1: "val1",
   parsed_message.key2: "val2"
}

The RenameKey Processor could then be used to get the desired output

Additional context
N/A

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the CopyValue processor and compare its behavior with the KeyValue processor's nested destination output. Determine how the requested to_root or object-flattening behavior should be configured, then verify that the example event produces key1 and key2 at the event root without losing the original message.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.