opensearch-project / opensearch-project/data-prepper
Create a decoder processor to decode Event keys
Open
@kkondaka is already working on this.
Since Dec 13, 2023.
enhancement
plugin - processor
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Is your feature request related to a problem? Please describe.
As a user of Data Prepper, my Events contain keys that are encoded in different formats, such as gzip, base64, and protobuf (https://protobuf.dev/programming-guides/encoding/).
Sample Event
{
"my_protobuf_key": "",
"my_gzip_key": "H4sIAAAAAAAAA/NIzcnJVyjPL8pJAQBSntaLCwAAAA==",
"my_base64_key": "SGVsbG8gd29ybGQ="
}
Describe the solution you'd like
A new processor called a decoder processor that can decode various encodings. The following configuration example would decode the three values in the example Event above
processor:
- decoder:
key: "my_base64_key"
# Can be one of gzip, base64, or protobuf
base64:
- decoder:
key: "my_gzip_key"
gzip:
- decoder:
key: "my_protobuf_key"
protobuf:
message_definition_file: "/path/to/proto_definition.proto"
Tasks
- #4016
- OTel decode processor
- Decode encoded data (base64)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Assessment
This issue has not been assessed yet.