opensearch-project / opensearch-project/data-prepper
Consolidate processor configurations for actions when there're key conflicts
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
These processors have similar overwrite_if_key_exists options:
- add_entries
- copy_values
- rename_keys
- parse_json
- parse_ion
- key_value
grok processor has a keys_to_overwrite option so that you can determine to overwrite or append (default) for each grok generated field.
There are 3 basic options when adding fields but the key already exists: overwrite, append or skip. We should consider making the options available and consistent throughout processors.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.
Additional context
See comments from https://github.com/opensearch-project/data-prepper/pull/4143
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the add_entries, copy_values, rename_keys, parse_json, parse_ion, key_value, and grok processors, along with the discussion in pull request 4143. Determine a consistent configuration for overwrite, append, and skip behavior across these processors. Done means the desired behavior and scope are agreed before implementation begins.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100