opensearch-project / opensearch-project/data-prepper

Create a model for binary data

Open
#4,096 0 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Is your feature request related to a problem? Please describe.

Data Prepper has sources which can pull binary data (mostly in base64) format. And we are adding some new processors which can decompress binary data. It would be good to handle binary data consistently so that we don't too much code spread across the project which will result in some processor combinations breaking a pipeline.

I'd like Data Prepper's sources and sinks to know their own encodings as much as possible.

Describe the solution you'd like

Create a new BinaryData model in data-prepper-api. Allow this to be set and retrieved from the Event model. This model can also be designed to avoid unnecessary encoding/decoding.

When a Data Prepper source gets binary data, it wraps it in the BinaryData model. Similarly, when writing to a sink use that same model.

There are some situations where the source cannot know the encoding. For example, JSON could have binary data encoded as base64 or base64. In such cases, the pipeline author will need to know the encoding and convert it accordingly.

class BinaryData {
  public byte[] getBinaryData();
  
  public static fromBase64Data(String base64) { ... }
}

There may also be an good way to decouple the binary data from the encoding itself.

Describe alternatives you've considered (Optional)

There may some useful third party libraries that have a similar solution we could make use of. Though, I'd still propose we keep our interface and use that for the internals.

Additional context

Coming from this comment: https://github.com/opensearch-project/data-prepper/issues/4016#issuecomment-1934998917

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the Event model and the data-prepper-api module to understand how values are represented and accessed. Trace how sources and sinks currently handle binary data, then define what BinaryData and Event integration must support and verify consistent handling across those boundaries.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend-api-design
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.