opensearch-project / opensearch-project/data-prepper

Supporting iterate_on for write_json

Open
#5,912 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.
When using write_json on a key with an array value, I would like to be able to convert each element in the array to its json string representation. Currently if an array is passed to write_json it will generate the json string representation of the array.

For example currently

"key1": [
{"a":"b", "c":"d"}
]

with the processor

- write_json:
    - source: key1
      target: some_key

outputs

"some_key": "[{\"a\":\"b\",\"c\":\"d\"}]"

But with iterate_on it would look something like this

"some_key": [
"{\"a\":\"b\",\"c\":\"d\"}",
]

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered (Optional)
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the write_json processor implementation and its tests, then compare them with the existing iterate_on behavior in other processors. Confirm how array inputs are currently serialized and define tests showing that iterate_on produces one JSON string per array element while preserving current behavior without it.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.