opensearch-project / opensearch-project/data-prepper

Support Limiting Array Entries in Hash-Map Values using Data Prepper Plugin

Open
#4,858 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement plugin - processor
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
I have JSON object in S3 with two fields like below and I want to limit the entries of those fields while uploading it to open search using ingestion pipeline.

"file" : { # file is a hash-map and keys in the file attribute are pre-defined
"read": ["value1","value2", ......"value1000"],
"write": ["value1","value2", ......"value1000"],
"delete": ["value1","value2", ......"value1000"],
}
"Scan": { # scan is also a hash-map and keys in the Scan attribute are dynamic and they are not fixed
"8080": ["1.2.3.4:8080", "5.6.7.8:8080", ..... "value1000"]
"450": ["1.2.34.5:450", ...."value1000"]
}
}

I want to limit the entries of the attributes(file.read, file.write, file.delete, scan.key1, scan.key2) to first 10 elements due to latency issues as elastic search has latency issues while querying large arrays.

Describe the solution you'd like
Need the data looks like below after processing using data prepper plugin

file.read: ["value1",... "value10"],
file.write: ["value1",... "value10"],
file.delete: ["value1",... "value10"],
scan.8080: ["value1",... "value10"],
scan.450: ["value1",... "value10"]

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No file, test, or specific plugin entry point is named; start by locating the Data Prepper plugin or processor responsible for transforming nested hash-map values. Done means the configured limit applies to arrays under both fixed and dynamic keys and produces the truncated fields shown in the request.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.