opensearch-project / opensearch-project/data-prepper
Support moving nested json to event root
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
The KeyValue processor writes its output maps to a destination field that is not the root of the event. As far as I'm aware there is not a way to move a nested field like that to the root of the event.
Example event
{message: "key1=val1 key2=val2"}
Current behavior of KeyValue processor
{
message: "key1=val1 key2=val2",
parsed_message: {
key1: "val1",
key2: "val2"
}
}
Desired output event
{
message: "key1=val1 key2=val2",
key1: "val1",
key2: "val2"
}
There are other cases beyond the KeyValue Processor where moving a nested field to the event root so building it into a processor would be useful
Describe the solution you'd like
The CopyValue processor should provide an optional parameter like to_root which can copy a field to the event root. It should also support a way of moving all fields within a json object
The configuration to get the desired output from above could look something like this:
processor:
- copy_values:
entries:
- from_object_key: "parsed_message"
to_root: true
Describe alternatives you've considered (Optional)
Data Prepper could support something like a Flatten Object processor to flatten nested json.
That config might look something like this
processor:
- flatten:
- key: "parsed_message"
And change the input to be like this
{
message: "key1=val1 key2=val2",
parsed_message.key1: "val1",
parsed_message.key2: "val2"
}
The RenameKey Processor could then be used to get the desired output
Additional context
N/A
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the CopyValue processor and compare its behavior with the KeyValue processor's nested destination output. Determine how the requested to_root or object-flattening behavior should be configured, then verify that the example event produces key1 and key2 at the event root without losing the original message.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100