opensearch-project / opensearch-project/data-prepper
Support extracting subLists from a list
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
As a user with a field in my Events that is a List, I would like to be able to turn that list into a subList
my_list: [ 0, 1, 2, 3, 4, 5, 6]
to
my_list: [ 4, 5, 6]
or
my_list: [ 0, 1, 2 ]
or
my_list: [ 2, 3, 4]
Describe the solution you'd like
I would like a new function support in Data Prepper expressions to convert to a subList
subList(<key>, <start_index, inclusive>, <end_index, exclusive>)
An end index value of -1, could signal to extract from start_index to the end of the list.
This function could then be used in the add_entries processor like this
add_entries:
entries:
- key: "my_list"
value_expression: '/subList(/my_list, 0, 2)'
Describe alternatives you've considered (Optional)
Truncate processor could support this instead
- truncate:
entries:
- source_keys: ["my_list"]
length: 3
start_at: 0
Additional context
Add any other context or screenshots about the feature request here.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the Data Prepper expressions implementation and the add_entries processor entry point. Review how existing expression functions handle list indexes, then define the inclusive-start and exclusive-end behavior, including an end index of -1, and add coverage for the requested subList examples.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100